Method and system for displaying holographic images within a real object

ABSTRACT

A system for displaying a holographic image of an object behind a real object surface, including a computing unit for computing data for displaying a three-dimensional image of an object, a location measurement unit for measuring a location of a surface of a real object, a display for displaying the three dimensional image of the object, wherein the computing unit is adapted to compute data to display the three-dimensional image of the object at least partly behind the surface of the real object. Related apparatus and methods are also described.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/078,639 filed on Aug. 22, 2018, which is a National Phase of PCTPatent Application No. PCT/IL2017/050225 having International filingdate of Feb. 22, 2017, which claims the benefit of priority under 35 USC§ 119(e) of U.S. Provisional Patent Application Nos. 62/353,718 filed onJun. 23, 2016 and 62/298,070 filed on Feb. 22, 2016. The contents of theabove applications are all incorporated by reference as if fully setforth herein in their entirety.

FIELD AND BACKGROUND OF THE INVENTION

The present invention, in some embodiments thereof, relates to a displaymethod and system which provides a 3D see-through vision view and, moreparticularly, but not exclusively, to a computer generated holographic(CGH) image display method and system which provides a 3D see-throughvision view, and even more particularly, but not exclusively, to a CGHimage display method and system which provides a display of a CGH imageof a real object with every point in the CGH image of the real objectaligned with its corresponding point in the real object in 3D space.

Additional background art includes:

U.S. Pat. No. 8,576,276 to Bar-Zeev et al; and

U.S. Patent Application Publication No. 2006/0176242 of Jaramaz et al.

The disclosures of all references mentioned above and throughout thepresent specification, as well as the disclosures of all referencesmentioned in those references, are hereby incorporated herein byreference.

SUMMARY OF THE INVENTION

The present invention, in some embodiments thereof, relates to a displaymethod and system which provides a 3D see-through vision view and, moreparticularly, but not exclusively, to a computer generated holographicimage display method and system which provides a 3D see-through visionview.

In some embodiments, a see-through vision display is provided of a firstobject within or behind a second object. In some embodiments, aholographic display system acquires and/or receives a 3D structure ofthe first object, including a location of a first marker on or in thefirst object, and the 3D structure of the second object and a secondmarker on or in the second object, and a relative position of the firstmarker relative to the second marker. The holographic display systemprojects an image of the first object inside or behind the second objectbased on detecting the markers and positioning the markers correctlyrelative to each other. In some embodiments the positions of the markersof the image of the first object are made to coincide with markers onthe second object. In some embodiments the positions of the markers ofthe image of the first object are made to be shifted by a known distanceand/or angle relative to markers on the second object.

The holographic image of an object provides both eye focus accommodationand eye convergence for a viewer, as natural distance/depth cues for aviewer's eye. The eye focus accommodation and eye convergence depth cuesof a holographic image are the same as provided by a real object.

According to an aspect of some embodiments of the present inventionthere is provided a method for displaying a holographic image of a bodyorgan at a correct location of the body organ, including obtaining afirst three-dimensional dataset including data for producing acomputer-generated-holographic (CGH) image of the body organ,determining a location of at least one first registration location inthe body organ, detecting a location of at least one second registrationlocation on the body, producing an interference based CGH image of thebody organ, and displaying the CGH image of the body organ, wherein thedisplaying the CGH image of the body organ includes displaying the CGHimage of the body organ so that the first registration location isdisplayed at a specific spatial location relative to the secondregistration location, and the CGH image of the body organ is alignedand located in a correct place of the body organ relative to the body.

According to some embodiments of the invention, the CGH image of thebody organ provides a viewer simultaneously with both eye convergenceand eye focus depth cues.

According to some embodiments of the invention, further includingobtaining a relative location and orientation of the second registrationlocation with respect to the first registration location.

According to some embodiments of the invention, further includingaccepting commands from a viewer of the CGH image via a user interfacein order to align the CGH image of the body organ to be located in acorrect place of the body organ in the body. According to someembodiments of the invention, the displaying the CGH image of the bodyorgan so that the CGH image of the body organ is aligned and located ina correct place of the body organ in the body is performed by acomputation unit aligning the first registration location to the secondregistration location, and displaying the CGH image of the body organaligned and located in a correct place of the body organ relative to thebody.

According to some embodiments of the invention, further includingobtaining a third three-dimensional dataset including data for producinga computer-generated-holographic (CGH) image of a tool, determining alocation of at least one third registration location on the tool,producing an interference based CGH image of the tool, and displayingthe CGH image of the tool, wherein the displaying the CGH image of thetool includes displaying the CGH image of the tool so that the thirdregistration location is displayed at a specific spatial locationrelative to at least one of the first registration location and thesecond registration location, so that the CGH image of the tool isaligned and located in a real location of the tool relative to the body,and the CGH image of the tool provides a viewer with both eyeconvergence and eye focus depth cues.

According to some embodiments of the invention, a first portion of thetool is invisible within the body, and at least a second portion of thetool including the third registration location is visible outside thebody.

According to some embodiments of the invention, the tool is an objectselected from a group consisting of a syringe, a needle, a robot arm, acatheter, an endoscope, and an image acquisition tool.

According to some embodiments of the invention, the tool is anultrasound imaging device for producing a three-dimensional dataset ofan inner portion of the body, and further including obtaining, from theultrasound imaging device, a fourth three-dimensional dataset includingdata for producing a computer-generated-holographic (CGH) image of a theinner portion of the body, producing an interference based CGH image ofthe inner portion of the body, and displaying the CGH image of the innerportion of the body, wherein the displaying the CGH image of the innerportion of the body includes displaying the CGH image of the innerportion of the body so that the CGH image of the inner portion of thebody is aligned and located in a correct place relative to theultrasound imaging device, and the CGH image of the inner portion of thebody provides a viewer with both eye convergence and eye focus depthcues.

According to some embodiments of the invention, the first registrationlocation is displayed at a same spatial location as the secondregistration location.

According to some embodiments of the invention, displaying the CGH imageof the body organ includes displaying the CGH image of the body organcorrectly oriented in space with a same orientation as the body organ isreally oriented in the body.

According to some embodiments of the invention, the first registrationlocation in the body organ includes a first group of registrationlocations including a plurality of registration locations in the bodyorgan.

According to some embodiments of the invention, the second registrationlocation on the body includes a second group of registration locationsincluding a plurality of registration locations on the body.

According to some embodiments of the invention, the location of thefirst registration location in the body organ is detectable by imageanalysis, and the detecting the location of the first registrationlocation in the body organ includes performing image analysis on thefirst three-dimensional dataset to detect the first registrationlocation in the body organ.

According to some embodiments of the invention, further includingproviding a first registration marker at the first registrationlocation, and wherein the first registration marker is detectable by animaging modality used for the obtaining the first three-dimensionaldataset.

According to some embodiments of the invention, the first registrationmarker has a shape which enables detection of orientation of the shapebased on a two-dimensional view of the shape.

According to some embodiments of the invention, wherein the firstregistration marker has a three-dimensional asymmetric shape.

According to some embodiments of the invention, wherein the firstregistration marker includes a surface designed to provide indication ofa 3D orientation of the first registration marker based on detecting thesurface by a sensor.

According to some embodiments of the invention, the providing the firstregistration marker includes attaching the first registration marker tothe body organ.

According to some embodiments of the invention, the location of thesecond registration location on the body organ is detectable by imageanalysis, and the detecting the location of the first registrationlocation in the body organ includes performing image analysis of animage of the body organ to detect the second registration location inthe body organ.

According to some embodiments of the invention, the detecting thelocation of the second registration location on the body includesperforming image analysis on an image of the body to detect the secondregistration location on the body.

According to some embodiments of the invention, further includingproviding a second registration marker at the second registrationlocation, and wherein the second registration marker is detectable byimage analysis. According to some embodiments of the invention, theproviding the second registration marker includes drawing a mark on thebody. According to some embodiments of the invention, the providing thesecond registration marker includes attaching the second registrationmarker to the body.

According to some embodiments of the invention, further includingobtaining a third three-dimensional dataset including data for producinga CGH image of a tool, detecting a location of a third registrationlocation on the tool, producing the CGH image of the tool, anddisplaying the CGH image of the tool, wherein the displaying the CGHimage of the tool includes displaying the CGH image of the tool so thatthe third registration location is displayed at a specific spatiallocation relative to the second registration location.

According to some embodiments of the invention, the location of thethird registration location in the tool is detectable by image analysis,and the detecting the location of the third registration location in thetool includes performing image analysis on an image of the tool todetect the third registration location in the tool.

According to some embodiments of the invention, further includingtracking movement of the third registration location in the toolrelative to the second registration location, displaying the CGH imageof the tool based, at least in part, on the tracking.

According to some embodiments of the invention, further includingproviding a third registration marker at the third registrationlocation, and wherein the third registration marker is detectable byimage analysis.

According to some embodiments of the invention, the providing the thirdregistration marker includes drawing a mark on the tool. According tosome embodiments of the invention, the providing the third registrationmarker includes attaching the third registration marker to the tool.

According to some embodiments of the invention, displaying the CGH imageof the body organ includes displaying by a head mounted CGH imagedisplay.

According to an aspect of some embodiments of the present inventionthere is provided a system for displaying a holographic image of a bodyorgan at a correct location of the body organ, including a computationunit for receiving a first three-dimensional dataset including data forproducing a computer-generated-holographic (CGH) image of a body organdetecting a location of a first registration location in the dataset ofthe body organ, and producing an interference based computer generatedhologram of the body organ, a sensor for detecting a location of asecond registration location on a body, and a CGH image display fordisplaying an interference based CGH image of the body organ using theinterference based computer generated hologram, wherein the displayingthe CGH image of the body organ includes displaying the CGH image of thebody organ so that the first registration location is displayed at aspecific spatial location relative to the second registration location,so that the CGH image of the body organ is aligned and located in acorrect place of the body organ in the body, and the CGH image of thebody organ provides a viewer with both eye convergence and eye focusdepth cues.

According to some embodiments of the invention, further including thecomputation unit configured to obtain a relative location andorientation of the second registration location with respect to thefirst registration location.

According to some embodiments of the invention, further includingmarkers that are detectable by an image acquisition system selected froma list consisting of Magnetic Resonance Imaging, ComputerizedTomography, Positron Emission Tomography-Computed tomography (PET-CT),nuclear imaging, X-ray, Infra-Red-camera, ultrasound, functionalimaging, metabolic imaging, Optical Coherence Tomography (OCT), andIntraVascular Ultrasound (IVUS) imaging.

According to some embodiments of the invention, the markers are selectedfrom a group consisting of a clip, a LED, an acoustic positioningsystem, an image pattern, a metallic pattern, an isotopic pattern, and atitanium pattern.

According to some embodiments of the invention, the markers have a shapewhich enables detection of orientation of the shape based on atwo-dimensional view of the shape.

According to some embodiments of the invention, the markers have athree-dimensional asymmetric shape. According to some embodiments of theinvention, the markers have a surface designed to provide indication ofa 3D orientation of the markers based on detecting the surface by asensor.

According to some embodiments of the invention, further includingmarkers for attaching to the body that are detectable by the sensor fordetecting a location of a second registration location on a body.

According to some embodiments of the invention, the sensor is selectedfrom a group consisting of a camera, an acoustic positioning system, andan electro-magnetic positioning system. According to some embodiments ofthe invention, the sensor is included in the CGH image display.

According to some embodiments of the invention, the markers are attachedto the body using at least one selected from a group consisting of ascrew, a pin, a clip, a metal fastener, a polymer fastener, a sticker,glue, and paint.

According to an aspect of some embodiments of the present inventionthere is provided a method for displaying a holographic image of a bodyorgan behind a body surface, including obtaining a firstthree-dimensional dataset including data for producing athree-dimensional image of a body organ, obtaining data describing alocation of the body organ relative to a display device for displaying athree-dimensional image, using the display device to display athree-dimensional image of the body organ to appear a specific distancebehind a surface of the actual body.

According to some embodiments of the invention, the display devicedisplays the three-dimensional image with a plurality of points in thethree-dimensional image each being in focus at a different distance fromthe display device.

According to some embodiments of the invention, at least some of thepoints in the three-dimensional image are in focus at a distance of lessthan 2 meters from the display device.

According to some embodiments of the invention, the display deviceincludes a Computer Generated Holographic (CGH) image display.

According to some embodiments of the invention, the image of the bodyorgan is aligned and located in a correct place of the body organrelative to the actual body. According to some embodiments of theinvention, when the display device is shifted relative to the actualbody, the display device maintains the three-dimensional image of thebody organ at a same location relative to the actual body.

According to some embodiments of the invention, when a distance of thedisplay device is changed relative to the actual body, the displaydevice changes focus of the three-dimensional image so that thethree-dimensional image of the body organ appears in focus at a samelocation relative to the actual body.

According to an aspect of some embodiments of the present inventionthere is provided a system for displaying a holographic image of a firstobject behind a real object surface, including a computing unit forcomputing data for displaying a three-dimensional image of a firstobject, a location measurement unit for measuring a location of asurface of a real object, a display for displaying the three dimensionalimage of the first object, wherein the computing unit is adapted tocompute data to display the three-dimensional image of the first objectat least partly behind the surface of the real object.

According to an aspect of some embodiments of the present inventionthere is provided a method for displaying an interference basedholographic image of a first object behind or within a visuallyobstructing second object, providing both eye convergence and eye focusaccommodation cues, including obtaining a first three-dimensionaldataset including data for producing a computer-generated-holographic(CGH) image of the first object, detecting a location of a firstregistration location in the first object, detecting a location of asecond registration location in the second object, producing the CGHimage of the first object, and displaying the CGH image of the firstobject, wherein the displaying the CGH image of the first objectincludes displaying the CGH image of the first object so that the firstregistration location in the first object is located at a specificspatial location relative to the second registration location.

According to some embodiments of the invention, the displaying the CGHimage of the first object includes displaying a continuous range of botheye convergence and eye focus accommodation cues.

According to some embodiments of the invention, the location of thefirst registration location in the first object is detectable by imageanalysis, and the detecting the location of the first registrationlocation in the first object includes performing image analysis on thefirst three-dimensional dataset to detect the first registrationlocation in the first object.

According to some embodiments of the invention, further includingproviding a first registration marker at the first registrationlocation, and wherein the registration marker is detectable by animaging modality used for the obtaining the first three-dimensionaldataset.

According to some embodiments of the invention, the providing the firstregistration marker includes producing a mark on the first object.According to some embodiments of the invention producing the markincludes drawing a mark on the first object. According to someembodiments of the invention, the providing the first registrationmarker includes attaching, affixing, fastening or inserting the firstregistration marker to the first object.

According to some embodiments of the invention, the location of thesecond registration location in the second object is detectable by imageanalysis, and the detecting the location of the second registrationlocation in the second object includes performing image analysis on animage of the second object to detect the second registration location inthe second object.

According to some embodiments of the invention, further includingproviding a second registration marker at the second registrationlocation, and wherein the second registration marker is detectable byimage analysis.

According to some embodiments of the invention, the providing the secondregistration marker includes drawing a mark on the second object.According to some embodiments of the invention, the providing the secondregistration marker includes attaching or inserting the secondregistration marker to the second object.

According to some embodiments of the invention, further includingobtaining a third three-dimensional dataset including data for producinga CGH image of a third object, detecting a location of a thirdregistration location in the third object, producing the CGH image ofthe third object, and displaying the CGH image of the third object,wherein the displaying the CGH image of the third object includesdisplaying the CGH image of the third object so that the thirdregistration location is displayed at a specific spatial locationrelative to the second location.

According to some embodiments of the invention, the location of thethird registration location in the third object is detectable by imageanalysis, and the detecting the location of the third registrationlocation in the third object includes performing image analysis on animage of the third object to detect the third registration location inthe third object.

According to some embodiments of the invention, further includingtracking movement of the third registration location in the third objectrelative to the second registration location, displaying the CGH imageof the third object based, at least in part, on the tracking.

According to some embodiments of the invention, further includingproviding a third registration marker at the third registrationlocation, and wherein the third registration marker is detectable byimage analysis.

According to some embodiments of the invention, the providing the thirdregistration marker includes drawing a mark on the third object.According to some embodiments of the invention, the providing the thirdregistration marker includes attaching the third registration marker tothe third object.

According to some embodiments of the invention, displaying the CGH imageof the first object includes displaying by a head mounted CGH imagedisplay.

According to some embodiments of the invention, displaying the CGH imageof the third object includes displaying by a head mounted CGH imagedisplay.

According to an aspect of some embodiments of the present inventionthere is provided a method for displaying an interference basedholographic image of an inner body organ within a body, providing botheye convergence and eye focus accommodation cues, including obtainingand/or receiving and/or using a first three-dimensional datasetincluding data for producing a computer-generated-holographic (CGH)image of the inner body organ, detecting a location of a firstregistration location in the inner body organ, detecting a location of asecond registration location on the body, producing the CGH image of theinner body organ, and displaying the CGH image of the inner body organ,wherein the displaying the CGH image of the inner body organ includesdisplaying the CGH image of the inner body organ so that the firstregistration location is displayed at a specific spatial locationrelative to the second registration location.

According to some embodiments of the invention, the displaying the CGHimage of the inner body organ includes displaying both eye convergenceand eye focus accommodation cues.

According to some embodiments of the invention, the location of thefirst registration location in the inner body organ is detectable byimage analysis, and the detecting the location of the first registrationlocation in the inner body organ includes performing image analysis onthe first three-dimensional dataset to detect the first registrationlocation in the inner body organ.

According to some embodiments of the invention, further includingproviding a first registration marker at the first registrationlocation, and wherein the first registration marker is detectable by animaging modality used for the obtaining the first three-dimensionaldataset.

According to some embodiments of the invention, the providing the firstregistration marker includes attaching the first registration marker tothe inner body organ.

According to some embodiments of the invention, the location of thesecond registration location in the body is detectable by imageanalysis, and the detecting the location of the second registrationlocation on the body includes performing image analysis on an image ofthe body to detect the second registration location on the body.

According to some embodiments of the invention, further includingproviding a second registration marker at the second registrationlocation, and wherein the second registration marker is detectable byimage analysis.

According to some embodiments of the invention, the providing the secondregistration marker includes producing a mark on the body. According tosome embodiments of the invention, the providing the second registrationmarker includes attaching the second registration marker to the body.

According to some embodiments of the invention the relative location ofthe first marker (seen by the image modality that acquires the 3D data)to the second marker (detected by the CGH projection unit) iscalculated, optionally by the CGH projection unit, in order to projectthe 3D data as a CGH image at the correct coordinates with respect tothe second marker.

In some embodiments one or more markers on an outside of a patient'sbody are detected both by the CGH projection unit and by the imagingmodality which acquires the 3D data, and the CGH image is projected suchthat the markers overlap.

In some embodiments a sensor outside the body detects location(s) of oneor more marker(s) inside a patient's body, and the imaging modalitywhich acquires the 3D data also detects the coordinates of the internalmarker(s), and the location(s) from the sensors are provided to the CGHprojection unit so as to display the CGH image, and such that the markerand its image overlap.

In some embodiments more than one sensor outside the body detectlocation(s) of one or more marker(s) inside a patient's body, and theimaging modality which acquires the 3D data also detects the coordinatesof the internal marker(s), and the location(s) from the sensors areprovided to the CGH projection unit so as to display the CGH image, andsuch that the marker and its image overlap.

In some embodiments a positioning system detects the marker(s)locations(s) by acoustic and/or electromagnetic sensors and sends thelocation(s) to the CGH image production system.

According to some embodiments of the invention, further includingobtaining a third three-dimensional dataset including data for producinga CGH image of a tool, detecting a location of a third registrationlocation associated with the tool, producing the CGH image of the tool,and displaying the CGH image of the tool, wherein the displaying the CGHimage of the tool includes displaying the CGH image of the tool so thatthe third registration location is displayed at a specific spatiallocation relative to the second location.

According to some embodiments of the invention, the location of thethird registration location in the tool is detectable by image analysis,and the detecting the location of the third registration location in thetool includes performing image analysis on an image of the tool todetect the third registration location associated with the tool.

According to some embodiments of the invention, further includingtracking movement of the third registration location in the toolrelative to the second registration location, displaying the CGH imageof the tool based, at least in part, on the tracking.

According to some embodiments of the invention, further includingproviding a third registration marker at the third registrationlocation, and wherein the third registration marker is detectable byimage analysis.

According to some embodiments of the invention, the providing the thirdregistration marker includes drawing a mark on the tool. According tosome embodiments of the invention, the providing the third registrationmarker includes attaching the third registration marker to the tool.

According to some embodiments of the invention, displaying the CGH imageof the inner body organ includes displaying by a head mounted CGH imagedisplay.

According to some embodiments of the invention, displaying the CGH imageof the tool includes displaying by a head mounted CGH image display.

According to an aspect of some embodiments of the present inventionthere is provided apparatus for displaying an interference basedholographic image of a first object behind or within a visuallyobstructing second object, providing both eye convergence and eye focusaccommodation cues, including a computation unit for obtaining a firstthree-dimensional dataset including data for producing acomputer-generated-holographic (CGH) image of the first object,detecting a location of a first registration location in the firstobject, producing the CGH image of the first object, a sensor fordetecting a location of a second registration location in the secondobject, and a CGH image display for displaying the CGH image of thefirst object, wherein the displaying the CGH image of the first objectincludes displaying the CGH image of the first object so that the firstregistration location in the first object is located at a specificspatial location relative to the second registration location.

According to an aspect of some embodiments of the present inventionthere is provided apparatus for displaying an interference basedholographic image of an inner body organ within a body, providing botheye convergence and eye focus accommodation cues, including acomputation unit for receiving a first three-dimensional datasetincluding data for producing a computer-generated-holographic (CGH)image of the inner body organ detecting a location of a firstregistration location in the inner body organ, and producing a computergenerated hologram of the inner body organ, a sensor for detecting alocation of a second registration location on the body, and a CGH imagedisplay for displaying the CGH image of the inner body organ, whereinthe displaying the CGH image of the inner body organ includes displayingthe CGH image of the inner body organ so that the first registrationlocation is displayed at a specific spatial location relative to thesecond registration location.

According to an aspect of some embodiments of the present inventionthere is provided a method for displaying an image of an object acquiredusing a first coordinate system by a CGH projection unit using a secondcoordinate system co-registered to the first coordinate system, themethod including: a. providing a CGH image projection unit that monitorsits display space, b. attaching to the object markers that aredetectable in both the first and the second coordinate systems, c.capturing an image of the object with the markers using the firstcoordinate system, d. detecting the markers by the CGH projection unitusing the second coordinate system, e. calculating a position of theobject in the second coordinate system, and f. projecting the CGH imageof the object at a location based on the position of the object in thesecond coordinate system.

According to some embodiments of the invention, the image taken usingthe first coordinate system includes markers that are detectable by anacquisition system selected from a list consisting of Magnetic ResonanceImaging, Computerized Tomography, PET-CT (Positron EmissionTomography-Computed tomography), nuclear imaging, X-ray,Infra-Red-camera, Ultrasound, functional imaging, metabolic imaging, OCT(Optical Coherence Tomography), IVUS (IntraVascular Ultrasound) imaging,Electrophysiology—electroanatomical mapping, and cone beam CT 3Drotational angiography.

According to some embodiments of the invention, the markers are selectedfrom a group consisting of a clip, a LED, an acoustic positioningsystem, an image pattern, a metallic pattern, an isotopic pattern, and atitanium pattern.

According to some embodiments of the invention, also markers that aredetectable by the CGH projection unit are attached to the body.

According to some embodiments of the invention, a sensor used by the CGHprojection unit to detect the marker is selected from a group consistingof a camera, an acoustic positioning system and an electro-magneticpositioning system.

According to some embodiments of the invention, the markers are attachedto the object using at least one selected from a group consisting of ascrew, a pin, a clip, a metal or polymer fastener, a sticker, glue andpaint.

According to an aspect of some embodiments of the present inventionthere is provided a method for co-registration of an image of an objectacquired at a first coordinate system to a CGH projection unit at asecond coordinate system including: a. providing a CGH projection unitthat monitors its display space, b. attaching to the object markers thatare detectable by the CGH projection unit in the second coordinatesystem, c. capturing the image of the object and the markers using thefirst coordinate system, d. sending the image of the object using thefirst coordinate system to the CGH projection unit, e. using the CGHprojection unit to detect the markers, f. calculating the position ofthe image of the object using the second coordinate system, and g.projecting the image of the object based on the calculating the positionof the image of the object using the second coordinate system.

According to some embodiments of the invention further including sendinga shape and location of an additional object to the CGH projection unit.

According to some embodiments of the invention further including: theCGH projection unit detecting the markers, the CGH projection unitcalculating a position of the object and the additional object in thesecond coordinate system; and projecting the image of the object and theimage of the additional object using the second coordinate system.

Unless otherwise defined, all technical and/or scientific terms usedherein have the same meaning as commonly understood by one of ordinaryskill in the art to which the invention pertains. Although methods andmaterials similar or equivalent to those described herein can be used inthe practice or testing of embodiments of the invention, exemplarymethods and/or materials are described below. In case of conflict, thepatent specification, including definitions, will control. In addition,the materials, methods, and examples are illustrative only and are notintended to be necessarily limiting.

Implementation of the method and/or system of embodiments of theinvention can involve performing or completing selected tasks manually,automatically, or a combination thereof. Moreover, according to actualinstrumentation and equipment of embodiments of the method and/or systemof the invention, several selected tasks could be implemented byhardware, by software or by firmware or by a combination thereof usingan operating system.

For example, hardware for performing selected tasks according toembodiments of the invention could be implemented as a chip or acircuit. As software, selected tasks according to embodiments of theinvention could be implemented as a plurality of software instructionsbeing executed by a computer using any suitable operating system. In anexemplary embodiment of the invention, one or more tasks according toexemplary embodiments of method and/or system as described herein areperformed by a data processor, such as a computing platform forexecuting a plurality of instructions. Optionally, the data processorincludes a volatile memory for storing instructions and/or data and/or anon-volatile storage, for example, a magnetic hard-disk and/or removablemedia, for storing instructions and/or data. Optionally, a networkconnection is provided as well. A display and/or a user input devicesuch as a keyboard or mouse are optionally provided as well.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Some embodiments of the invention are herein described, by way ofexample only, with reference to the accompanying drawings. With specificreference now to the drawings in detail, it is stressed that theparticulars shown are by way of example and for purposes of illustrativediscussion of embodiments of the invention. In this regard, thedescription taken with the drawings makes apparent to those skilled inthe art how embodiments of the invention may be practiced.

In the drawings:

FIG. 1A is a view of a head with a Head Mounted Display (HMD) accordingto an example embodiment of the invention;

FIG. 1B is a simplified illustration of an example embodiment of adisplay on an adjustable arm according to an example embodiment of theinvention;

FIG. 2A is a simplified illustration of a HMD displaying a holographicimage of a first object and a visually obstructing second object,according to an example embodiment of the invention;

FIG. 2B is a simplified functional illustration of a display systemaccording to an example embodiment of the invention;

FIGS. 2C, 2D and 2E are simplified illustrations of a HMD displaying aholographic image of a first object maneuvered to appear within avisually obstructing second object, according to an example embodimentof the invention;

FIG. 3 is a simplified illustration of two HMDs displaying a sameholographic image of a first object behind or within a visuallyobstructing second object, according to an example embodiment of theinvention;

FIGS. 4A and 4B are simplified illustrations of a HMD displaying aholographic image of a first object behind or within a visuallyobstructing second object, as well as an additional object at leastpartially behind or within the visually obstructing second object,according to an example embodiment of the invention;

FIG. 4C is a simplified illustration of a HMD displaying a holographicimage of a first object, obtained and registered in real time, behind orwithin a visually obstructing second object, as well as an additionalobject at least partially behind or within the visually obstructingsecond object, also obtained and registered in real time, and one ormore additional guide lines, according to an example embodiment of theinvention;

FIG. 4D is a simplified illustration of a HMD displaying a holographicimage of a first object, obtained and registered in real time, behind orwithin a visually obstructing second object, as well as an additionalobject at least partially behind or within the visually obstructingsecond object, also obtained and registered in real time, according toan example embodiment of the invention;

FIG. 4E is a simplified functional illustration of a display system fordisplaying a CGH image of ultrasound data according to an exampleembodiment of the invention;

FIG. 4F is a simplified illustration of a HMD displaying a holographicimage of a first object and a visually obstructing second object,according to an example embodiment of the invention;

FIG. 4G is a simplified illustration of a HMD displaying a holographicimage of a first object behind or within a visually obstructing secondobject, as well as an additional object at least partially behind orwithin the visually obstructing second object, according to an exampleembodiment of the invention;

FIG. 4H is a simplified illustration of an imaging system for obtainingthree-dimensional data for displaying a 3D image of internal organs in abody, according to an example embodiment of the invention;

FIG. 4I is a simplified illustration of an imaging system for obtainingthree-dimensional data for displaying a 3D image of internal organs in abody, according to an example embodiment of the invention;

FIG. 4J is a simplified illustration of an imaging system and anadditional object, according to an example embodiment of the invention;

FIG. 5A is a simplified illustration of a HMD displaying a holographicimage of a first object behind or within a visually obstructing secondobject, according to an example embodiment of the invention;

FIGS. 5B and 5C are simplified illustrations of a specific portion ofFIG. 5A, in which a HMD is displaying a holographic image of pipesbehind a visually obstructing wall, according to an example embodimentof the invention;

FIG. 6A is a simplified line drawing of blood vessels and metastasislocations according to an example embodiment of the invention;

FIG. 6B is a simplified line drawing of the blood vessels and themetastasis locations of FIG. 6A according to an example embodiment ofthe invention;

FIG. 7A is a simplified isometric line drawing illustration of needlesand a tumor with a specific body volume according to an exampleembodiment of the invention;

FIG. 7B is a simplified isometric line drawing illustration of needlesand a tumor with a specific body volume according to yet another exampleembodiment of the invention;

FIG. 8A is a simplified flow chart illustration of a method fordisplaying an interference based holographic image of an inner bodyorgan within a body, providing both eye convergence and eye focusaccommodation cues, according to an example embodiment of the invention;

FIG. 8B is a simplified flowchart illustration of a method fordisplaying a holographic image of a body organ at a correct location ofthe body organ within a body, according to an example embodiment of theinvention;

FIG. 9 is a simplified flow chart illustration of a method fordisplaying an interference based holographic image of a first objectbehind or within a visually obstructing second object, providing botheye convergence and eye focus accommodation cues, according to anexample embodiment of the invention;

FIG. 10 is a simplified block diagram illustration of apparatus fordisplaying an interference based holographic image of a first objectbehind or within a visually obstructing second object, providing botheye convergence and eye focus accommodation cues, according to anexample embodiment of the invention;

FIG. 11 is a simplified flow chart illustration of a method fordisplaying an image of an object acquired using a first coordinatesystem by a CGH projection unit using a second coordinate systemco-registered to the first coordinate system according to an exampleembodiment of the invention; and

FIG. 12 is a simplified flow chart illustration of a method forco-registration of an image of an object acquired at a first coordinatesystem to a CGH projection unit at a second coordinate system accordingto an example embodiment of the invention.

DESCRIPTION OF SPECIFIC EMBODIMENTS OF THE INVENTION

The present invention, in some embodiments thereof, relates to a displaymethod and system which provides a 3D see-through vision view and, moreparticularly, but not exclusively, to a computer generated holographic(CGH) image display method and system which provides a 3D see-throughvision view, and even more particularly, but not exclusively, to a CGHimage display method and system which provides a display of a CGH imageof a real object with every point in the CGH image of the real objectaligned with its corresponding point in the real object in 3D space.

Overview

An aspect of some embodiments of the invention involves displaying oneor more computer generated image(s) of a first object behind a surfaceof a real, actual second object.

In some embodiments the displaying is of a computer generated image of abody organ behind the real body surface.

In some embodiments the displaying is of a Computer GeneratedHolographic (CGH) image of the first object behind the real bodysurface.

In some embodiments the three-dimensional image is displayed by, forexample, an augmented reality display such as a head mounted display,which also enables a viewer to view the real object in the real world.The image of the first object is displayed in a same space as the realobject is viewed, so the viewer sees both the image and the real object.In some embodiments the viewer sees the image, for example of aninternal organ, apparently behind a surface of the real object, forexample apparently beneath the skin of a patient whose internal organ isbeing displayed.

In some embodiments the image of the internal organ is located at itscorrect location relative to a patient's body, optionally by aligningknown landmark(s) on the body, such as a natural shape of the body orsuch as a marking added to the body, with known landmarks in theinternal organ, again a natural shape of the internal or a marking addedto the organ.

An aspect of some embodiments of the invention involves displaying oneor more Computer Generated Holographic (CGH) image(s) in a space of anexisting object. The CGH image(s) then, in some cases, appears to bevisible inside the existing object, or the existing object, in somecase, appears to be inside the CGH image(s).

An aspect of some embodiments of the invention involves displaying oneor more CGH images in a space of an existing object.

In some embodiments the CGH image(s) are optionally images of internalbody organs, and the real object is a body containing the real internalorgans. Such embodiments are potentially useful in medicalcircumstances, where a physician is displayed a medical image in itscorrect location, and does not have to shift his view between looking ata patient and looking at the patient's medical images.

In some embodiments the CGH image(s) are optionally images of hiddenelements of construction, and the real object may be walls, ceiling orfloor hiding the elements.

An aspect of some embodiments of the invention involves aligning a CGHimage of internal or hidden elements to an existing object so that theCGH image of the elements appears in its correct location and/or sizeand/or orientation relative to the existing object.

In some embodiments, the alignment is performed by a viewer usingcontrol commands to a display system displaying the CGH image(s) inorder to align the elements with the exiting object. By way of a medicalexample, a physician commanding the CGH display to shift and/or rotateand/or scale a CGH image of a patient's internal organs to the patient'sbody. By way of a construction example, a viewer commanding the CGHdisplay to shift and/or rotate and/or scale a CGH image of wiring to thevisible wiring outlets on a wall.

In some embodiments, the alignment is performed automatically by acomputer controlling the CGH display to shift and/or rotate and/or scalea CGH image to align markers detectable in a dataset for producing theCGH image with markers detectable in the existing object. By way of amedical example, aligning detectable elements of a patient's anatomywithin the CGH image to detectable elements viewable on the patient'sbody. By way of a construction example, automatically computing valuesfor the CGH display so as to shift and/or rotate and/or scale a CGHimage of wiring to visible wiring outlets on a wall.

An aspect of some embodiments of the invention involves performingvarious medical procedures using displayed CGH image(s) of internalorgans and/or medical tools aligned to a patient's body, enablingviewing the internal organs, the tool(s) and the patient's body all intheir real place. Various embodiments are described below which teachbeneficial uses of viewing internal organs and/or tools in their correctlocation in a patient's body.

An aspect of some embodiments of the invention involves using detectablemarkers in order to align the CGH image to the existing object.

In some embodiments two or more CGH images are optionally superposed toappear one inside the other.

In some embodiments displaying, to a viewer, two or more real and/or CGHimages in a same space, at least up to an appropriate degree ofaccuracy.

The terms see-through vision or see-through view are used herein torefer to displaying a superposition of a CGH image and a real object, ortwo or more CGH images, in a same space/volume of space, or displayingof one or more CGH images behind a real object.

The term see-through vision may also be termed pseudo X-ray vision, asmay be understood from the description above and the examples providedherein.

An aspect of some embodiments of the invention involves fusing views ofreal images with holographic images of real objects, optionally producedfrom three-dimension (3D) datasets describing the real objects, and/orwith holographic images of computed objects or imaginary objectsproduced from three-dimensional (3D) datasets describing the computed orimaginary real objects.

The term “display” is used throughout the present specification andclaims to mean a Computer Generated Holographic (CGH) display such as,by way of some non-limiting examples, a “head mounted display” (HMD) asdepicted in FIG. 1A and a display on an adjustable arm as depicted inFIG. 1B.

An aspect of some embodiments of the invention involves detectinglocations visible in a real objects and also visible or detectable in a3D image or dataset of the real object, and aligning also termedregistering, the CGH image of the object with the real object in 3Dspace.

In some embodiments the registering is performed as a registering of a3D CGH image to a real body.

In some embodiments the registering is performed as a registering ofmore than one 3D CGH image to a real body, for example when the 3D CGHimages are of the same real body.

In some embodiments the registering is performed as a registering of a3D CGH image to another 3D image of the same real body.

In some embodiments the registering is performed across several realbodies and/or several CGH images. By way of a non-limiting example aregistering of a first 3D CGH image to a 3D image of a first real body,and a registering of a second 3D image to the first real body and/or tothe first CGH image. By way of another non-limiting example, a 3D CGHimage of a patient's body is registered with the patient's body, and animage of a surgical tool partway inside the patient's body is registeredwith a portion of the surgical tool which is visible outside thepatient's body. The above example, potentially enables a surgeon to viewinternal organs within a patient's body, and to view and image his/hersurgical tool in its correct location within the body. Such an exampleprovides see-through vision of objects and organs which are invisible toa naked eye.

In some example embodiments a physician steers a 3D CGH image of a bodyand/or internal organs to a correct, aligned location within the realbody. The steering may require moving the image in one, two or threedimensions, and/or rotating the image, and/or scaling the image.

In some embodiments alignment in two dimensions is performed by thephysician aligning known locations in the CGH image of the body withknown locations of the real body. In some embodiments alignment in adepth dimension is optionally performed by the physician using 3D depthcues such as eye focus accommodation and/or eye convergence to align theCGH image display of the body with the actual location of the real body.

In some embodiments a marker detection system detects markers on or inthe real body, and the CGH image display system, which also detectslocations of the markers in the 3D CGH image of the body and/or internalorgans, aligns the CGH image correctly with the real body.

In some embodiments a surgeon can keep looking at a patient's body andsee medical images drawn to a correct scale and correctly aligned to thepatient's body.

In some example embodiments it is not a physician which steers a 3D CGHimage of a body and/or internal organs to a correct, aligned locationwithin the real body, but some other scenario, such as a viewer viewingpipes and/or wiring within or behind a wall, and aligning thepipes/wiring correctly with their true location in the wall, optionallybased on common details visible on the real wall and also in the CGHimage of the wall.

Before explaining at least one embodiment of the invention in detail, itis to be understood that the invention is not necessarily limited in itsapplication to the details of construction and the arrangement of thecomponents and/or methods set forth in the following description and/orillustrated in the drawings and/or the Examples. The invention iscapable of other embodiments or of being practiced or carried out invarious ways.

Reference is now made to FIG. 1A, which is a view of a head 102 with aHead Mounted Display (HMD) 105 according to an example embodiment of theinvention. FIG. 1A depicts an isometric view of a HMD 105 allowing aviewer to see through the display while also displaying a holographicimage. In some embodiments the HMD 105 may be implemented as eyeglassesor goggles.

It is noted that a holographic image, by way of a non-limiting example aComputer Generated Holographic (CGH) image, displays an image whichprovides true depth cues. For example, a holographic image provides athree-dimensional image for which a human viewer's eye potentiallyperforms both eye focus accommodation, and eye convergenceaccommodation.

In contrast, for example, a stereoscopic display provides an illusion ofdepth by displaying two slightly shifted images, one to each eye.

In addition, for example, an isometric image display provides neithereye convergence depth cues, nor eye focus depth cues, only an illusionof a three-dimensional image by virtue of displaying an isometric image.

Reference is now made to FIG. 1B, which is a simplified illustration ofan example embodiment of a display on an adjustable arm according to anexample embodiment of the invention. FIG. 1B depicts a system 130 whichincludes an optional enclosure 134 to optionally include a computingunit; as well as optical components such as coherent light sources,SLM(s), and optionally additional optical components for displaying aCGH image. FIG. 1B also depicts an optional adjustable arm 132 from thesystem mounting surface (not shown) to enclosure 134 and an optional arm136 from the enclosure 134 to a semi-transparent/semi-reflectiveeyepiece 138, optionally containing additional optical components fordisplaying a CGH image. In some embodiments thesemi-transparent/semi-reflective eyepiece 138 is apartially-transmitting/partially-reflecting curved mirror at projectedlight wavelengths, or a volumetric holographic element which acts as aneyepiece. FIG. 1B also depicts one or more optional handle(s) 142. FIG.1B also depicts a viewer 140 using the system 130.

In some embodiments the adjustable arm 132 is a boom arm.

An aspect of some embodiments of the invention involves displaying, byway of a non-limiting example, a CGH image of a body's internal organs,correctly placed within a view of the real body. In some embodiments,such displaying is optionally achieved by the display device detecting alocation of the real body, and displaying the CGH image of the body'sinternal organs at their correct location.

The terms “correct” place and/or correct orientation are used herein tomean a degree of accuracy appropriate with a medical need or a medicalprocedure.

The terms “correct location” “correct place” “correct coordinates”“correct orientation” “real location” “real place” “real coordinates”“real orientation” “same location” “same place” “same coordinates” “sameorientation” in all their grammatical forms are used throughout thepresent specification and claims interchangeably to mean that a firstCGH image is aligned with a second CGH image or with a second realobject such that points on the first CGH image appear to a viewer in asame location in space as corresponding points in the second CGH imageor in the second real object, to the above-mentioned appropriate degreeof accuracy.

In some embodiments, such as performing a medical diagnosis, the CGHimage is optionally approximately aligned with a patient's body and/orvisible features on or within the patient's body, and the CGH imageserves as an in-place image, showing a physician the internal organs intheir approximate location. By way of a non-limiting example, such adisplay potentially eliminates a need for a physician to perform featsof mental imagery to imagine the internal organs in place, potentiallyeliminating, for example, mistakes between left and right of thepatient's body in identifying or treating organs.

In some embodiments, such as performing a surgical procedure, the CGHimage is optionally aligned accurately, and the CGH image potentiallyserves, by way of a non-limiting example, for determining a location ofa tool relative to an internal organ, optionally as part of the surgicalprocedure.

In some embodiments the accuracy of alignment is measured as accuracy ofwithin 10 pixels or voxels of the CGH image to an actual body, and anaccuracy of less than 10 pixels/voxels, such as 5, 2, 1 pixels/voxels,and even sub-pixel accuracy, such as 0.5, 0.2, 0.1 pixels/voxels orbetter.

In some embodiments the accuracy of alignment is measured as accuracy ofwithin 10 millimeters of the CGH image to an actual body, and anaccuracy of less than 10 millimeters, such as 5, 2, 1 millimeters, andeven sub-millimeter accuracy, such as 0.5, 0.2, 0.1 millimeters orbetter.

In some embodiments the above-mentioned accuracy of alignment isachieved for a CGH image which is displayed at an apparent distance of15-200 centimeters from a viewer's eyes. In some embodiments theabove-mentioned accuracy of alignment is achieved for a CGH image whichis displayed at an apparent hand-reach distance from a viewer.

In some embodiments sub-pixel accuracy of alignment is optionallyachieved by image processing a dataset for producing the CGH image andan image of the patient obtained by the display system, and minimizingmean error between a shape of the body of the patient in the CGH imageand a shape of the body of the patient in the image of the patientobtained by the display system.

In some embodiments sub-pixel accuracy of alignment is optionallyachieved by detecting markers in the dataset for producing the CGH imageand detecting the markers in an image of the patient obtained by thedisplay system, and minimizing mean error between locations of themarkers in the CGH image and in the image of the patient obtained by thedisplay system.

In some embodiments the accuracy of alignment/registration is achievedin one or more of the following measures: translation alongthree-dimensions, such as sideways, up-down- and along the depthdirection; scaling; and rotation.

The term “co-registration” is used throughout the present specificationand claims to mean aligning a displayed CGH image with a realobject/body, and/or with an additional displayed CGH image(s).

In some embodiments, the co-registration aligns the displayed CGH imagewith a real object/body, in such a way that a viewer's eye convergenceand eye focus for the displayed CGH image is identical to the viewer'seye convergence and eye focus for viewing the real object. In someembodiments the viewer's eye convergence and eye focus is identical evenwhen the real object is hidden, and the displayed image is visible atthe location of the real object.

Reference is now made to FIG. 2A, which is a simplified illustration ofa HMD displaying a holographic image of a first object and a visuallyobstructing second object, according to an example embodiment of theinvention.

FIG. 2A depicts the holographic image 210 of the first object, forexample internal organs, for example liver and related blood vessels,behind and/or within the visually obstructing second object, for examplethe skin of an actual body of the patient whose internal organs aredisplayed.

FIG. 2A depicts a viewer 201 wearing a HMD 202, which displays theholographic image 210 of the internal organs of a patient.

The viewer 201 also sees the body 211 of the patient, so the viewer 201sees the internal organs (liver and related blood vessels) within thebody 211.

In some embodiments the HMD 202 displays to the viewer 201 just portionsof body organs, so the viewer 201 sees the portions of the body organsappear within the body 211 at their correct locations.

In some embodiments the HMD 202 displays to the viewer 201 artificialdevices implanted in the body 211 of the patient, such as stent(s),heart valves, pacemakers, artificial joints, and so on, so the viewer201 sees the artificial devices appear within the body 211 at theircorrect locations.

In some embodiments the viewer 201 also sees artificial devicesimplanted in the body 211 of the patient, such as stent(s), heartvalves, pacemakers, artificial joints, and so on, so the viewer 201 seesthe artificial devices appear within the body 211 at their correctlocations.

In some embodiments, the liver and related blood vessels are correctlylocated in space relative to the body 211. In some embodiments thedisplay system, the HMD 202 for example, obtains a first relativelocation or coordinates of a location or locations 206 a 206 b on thebody 211 of the patient, relative to the HMD 202, and also obtainssecond relative location(s) or coordinates of the location(s) 206 a 206b relative to a three-dimensional (3D) imaging dataset used forproducing the holographic image 210 of the internal organs of thepatient.

In some embodiments the locations 206 a 206 b may be body parts, suchas, by way of a non-limiting example nipples, or an umbilicus, or bodycontours or features which can be identified by the HMD 202 as well asbe identifiable in the (3D) imaging dataset, either by computerizedimage processing and/or by the viewer 201 viewing the locations 206 a206 b in the CGH image as well as in a real view.

In some embodiments the locations 206 a 206 b may be artificialmarkings, such as described in more detail elsewhere in the herein.

In some embodiments, the HMD 202 displays the holographic image 210 ofthe internal organs so that the locations 206 a 206 b in the holographicimage coincide, also termed herein co-register, with the locations 206 a206 b in the real world, on the body 211 of the patient. Suchco-registration potentially displays the internal organs in the correctlocation relative to the body 211.

In some embodiments, the HMD 202 also displays a skin 206 c of thepatient. In some embodiments, the skin 206 c is displayed based on datafrom a three-dimensional (3D) imaging dataset used for producing theholographic image 210 of the internal organs of the patient.

In some embodiments the HMD 202 optionally has one or more sensor(s) 204which can detect and locate the markings 206 a 206 b. The sensor(s) 204optionally measures distance and/or angle toward the markings 206 a 206b on the patient's body 211, for example along lines 205 a 205 b. Themeasurement potentially enables the HMD 202 to determine a location ofthe patient's body 211 relative to the displayed CGH image 210 of theinternal organs.

The sensor(s) 204 may optionally be any one of the sensors describedherein.

The locations 206 a 206 b may optionally be any of markings describedherein.

In some embodiments, the display, such as the HMD 202, is a see-throughdisplay, such that a viewer sees the real world through the display atthe same time as seeing the CGH image 210. Such a display is termed anaugmented reality display, since both reality e.g. body 211 and the CGHimage 210 are seen.

When the CGH image 210 is an image of a real object, e.g. of a bodyorgan, the display may optionally be termed a fused reality display.

In some embodiments, knowing the location of the internal organsrelative to the body 211 and/or relative to HMD 202 potentially enablesthe HMD 202 to implement a man-machine-interface (MMI).

By way of some non-limiting examples, a MMI command may optionallyinclude aligning a CGH image of the internal organs to the body, and mayoptionally include a measure of accuracy at which the alignment shouldbe done.

By way of some non-limiting examples, a MMI command may optionallyinclude to display a CGH image of the internal organs at some locationother than aligned to the body such as, by way of some non-limitingexamples: floating above the body, floating above the body and rotatedby a specific angle relative to the body or in proximity to the body.

By way of some non-limiting examples, a MMI command may optionallyinclude to display a CGH image of the internal organs re-aligned, backfrom the not-aligned location.

In some embodiments, when a viewer 201 wearing the HMD 202 moves hishead, and/or moves his eyes to a different direction, the HMD 202compensates for the movement, and displays the CGH image 210 in itscorrect location, registered to the body 211.

In some embodiments, when the body 211 of a patient moves or shifts, theHMD 202 compensates for the movement, optionally by tracking thelocations 206 a 206 b, and displays the CGH image 210 in its correctlocation, registered to the body 211.

It is noted that the viewer 201 moving the HMD potentially enables theviewer to inspect or view the CGH image from different angles, the sameas inspecting or viewing the real object which the CGH image isdisplaying.

Such tracking of the viewer 201 movements and/or patient movements aredescribed with reference to FIG. 2A, yet it is to be understood by aperson skilled in the art that such tracking applies to all embodimentsdescribed herein.

Reference is now made to FIG. 2B, which is a simplified functionalillustration of a display system according to an example embodiment ofthe invention.

FIG. 2B shows functional blocks in an example embodiment of theinvention and how they optionally interact in order to display aholographic image of a body organ, optionally at a correct location ofthe body organ within a body.

FIG. 2B depicts a body 211.

An imaging system (not shown) optionally images the body 211, optionallyproducing a first three-dimensional dataset 231 for producing acomputer-generated-holographic (CGH) image of a body organ 210.

The dataset 231 is optionally fed into a computation unit 233, which canproduce a computer-generated hologram of the body organ. In someembodiments one or more marker(s) location(s) and/or orientation(s) areoptionally part of the data set.

A sensor 234 optionally detects locations of features in the body 211which may optionally serve to align an image of the body organ with aview of the real body 211. In some embodiments the features are naturalfeatures such as nipples 206 a 206 b. In some embodiments the featuresare markers 212 a 212 b on and/or in the body. The sensor 234 optionallyprovides data, for example distance and direction from the sensor to thefeatures or markers, to the computation unit 233, which can use the datato calculate and produce a computer generated hologram for displayingthe CGH image of the body organ so that the CGH image of the body organ210 is aligned and located in a correct place of the body organ in thebody 211. In some embodiments one or more markers are optionallydetected by the computation unit 233.

In some embodiments a sensor provides an inner marker location, or arelative position of the inner marker with respect to an outer marker,and the CGH unit senses the location of the outer marker, and optionallycalculates the location of the inner marker.

Reference is now made to FIGS. 2C, 2D and 2E, which are simplifiedillustrations of a HMD displaying a holographic image of a first objectmaneuvered to appear within a visually obstructing second object,according to an example embodiment of the invention.

FIG. 2C depicts a holographic image 210 of the first object, for examplea torso 203 including internal organs 210, for example liver and relatedblood vessels of a patient, floating in the air. FIG. 2C also depicts aviewer 201 wearing a HMD 202, which displays a field of view 219, whichdisplays the holographic image 210. The holographic image is optionallyfloating in the air, somewhere in a vicinity of a patient's body (notshown in FIG. 2C, but shown in FIGS. 2D and 2E).

FIG. 2D depicts the holographic image 210 of the torso 203 and theinternal organs 210 floating in the air, and maneuvered by the viewer201 toward a patient's body 211, also showing a torso 203 b. FIG. 2Ddepicts the holographic image 210 partly inside and partly outside ofthe patient's body 211.

In some embodiments the viewer 201 provides user commands to a userinterface associated with the HMD 202, to move a location of the displayof the CGH image 210 in space, by way of a non-limiting example toward acorrect location for the internal organs within the patient's body 211.

In some embodiments the viewer 201 provides user commands to a userinterface associated with the HMD 202, to rotate the display of the CGHimage 210 in space, by way of a non-limiting example to appear from acorrect aspect or angle for the internal organs within the patient'sbody 211.

In some embodiments the viewer 201 provides user commands to a userinterface associated with the HMD 202, to scale (change size) thedisplay of the CGH image 210 in space, by way of a non-limiting exampleso as to appear at a correct size for the internal organs relative tothe patient's body 211.

In some embodiments the maneuvering of the holographic image 210 isoptionally performed by a user interface, the viewer 201 maneuvering theimage 210 in three-dimensions in space (up-down, left-right, andforward-backward in a depth direction relative to the viewer),optionally rotating around three axes (optionally up-down, left-right,and forward-backward in a depth direction relative to the viewer), andoptionally scaling (making the holographic image 210 larger or smaller).

In some embodiments the viewer 201 uses depth cues provided by theholographic image 210 and by a viewable portion of the real object whichis the body of the patient 211 in order to maneuver the holographicimage 210 in the depth direction.

In some embodiments the viewer aligns natural registration markings,such as, by way of some non-limiting examples, a patient's 211 nipples221 and/or the outline of the patient's torso 203 in the holographicimage 210 and a patient's nipples 221 b and/or the outline of thepatient's torso 203 b in the real patient's 211 body.

FIG. 2E depicts the holographic image 210 of the internal organs behindor within the visually obstructing skin of the actual body 211 of thepatient whose internal organs are displayed.

In some embodiments, the location of the display of the CGH image 210 inspace is automatically located and/or rotated and/or scaled to appear ata correct location for the internal organs within the patient's body211. By way of a non-limiting example, the HMD 202 locates identifiablefeatures which are common to both the visible body 211 and to a datasetused for producing the CGH image of the internal organs, and producesthe display of the internal organs so that the identifiable features areco-registered. By way of a non-limiting example the identifiablelocations may be nipples 221 and/or an umbilicus on the body 211, or anoutline of the body 211, or markers as described elsewhere herein.

In some embodiments, the display of the CGH image 210 in space isoptionally manipulated, by a user or automatically, with one or more ofthe following image manipulations: rotation; shifting/moving in space;scaling and/or zooming; slicing an image; fusing two images; and markinga location in an image.

In some embodiments the identifiable features which are common to boththe visible body 211 and to the dataset include bony protuberances,which are less likely to shift relative to the body when a patientshifts her/his position, and less likely to shift relative to internalorgans of the body when the patient shifts her/his position.

In some embodiments a physician thus steers a 3D CGH image of a bodyand/or internal organs to a correct, aligned location within the realbody. In some embodiments alignment in two dimensions is performed bythe physician aligning known locations in the CGH image of the body withknown locations of the real body. In some embodiments alignment in adepth dimension is optionally performed by the physician using 3D depthcues such as eye focus accommodation and/or eye convergence to align theCGH image display of the body with the actual location of the real body.

In some embodiments, the MMI may optionally be as described inabove-mentioned U.S. Pat. No. 8,500,284; and/or as in U.S. PatentApplication Publication Number 2014/0033052; and/or as in PCT PatentApplication Publication WO 2015/004670.

In some embodiments, tracking a body's location and/or orientation inspace relative to the HMD 202 is optionally performed by an externalsystem tracking the body 211 and/or the HMD 202.

In some embodiments, tracking a display's orientation in space, such asthe HMD 202, is optionally performed by the display itself, by opticallytracking location of objects, external to the display, in space; byoptically tracking specific markings in a vicinity of the display inspace; by using direction finding similarly to direction finding bymobile devices such as smart phones; by using an accelerometer; by usinga gravity sensor; and in case of a display mounted on an adjustable arm,optionally measuring angles of sections of the adjustable arm.

In some embodiments, a tracking system for determining three-dimensionalcoordinates is optionally an optical tracking system monitoring objectsin a same space as the CGH image is displayed.

In some embodiments, a tracking system for determining three-dimensionalcoordinates is optionally an electromagnetic tracking system monitoringobjects such as markers in a same space as the CGH image is displayed.

In some example embodiments, a view of internal organs within a body, byway of a non-limiting example as described with reference to FIG. 2A,may be done, by way of a non-limiting example, for planning a surgicalprocedure, during a surgical procedure, and for teaching purposes.

In some example embodiments, a view of a first object within and/orbehind a second object may be done, by way of a non-limiting example,for displaying pipes behind or within a wall, pipes underneath theground, geologic formations beneath the ground, electric wires behind orwithin walls, a sewer pipe beneath the ground, a cable in a pipe, and soon.

An aspect of some embodiments of the invention involves displaying, byway of a non-limiting example, a CGH image of a body's internal organs,correctly placed within a view of the real body, via two or moredisplays, each one of the displays displaying the same CGH image, asviewed from a viewpoint of an associated CGH image display, of thebody's internal organs, correctly placed within the view of the samereal body, to the above-mentioned appropriate degree of accuracy.

In some embodiments, such displaying is optionally achieved by each oneof the display devices detecting a location of the real body, anddisplaying the CGH image of the body's internal organs at the correctlocation and from a correct viewpoint. It is noted that a CGH displays atrue depth at CGH image points, causing eye focus accommodation and eyeconvergence, the CGH image appears as the real object would appear to aviewer with reference to eye focus accommodation and eye convergence.

In some embodiments, such displaying is optionally achieved by one ofthe display devices detecting a location and/or orientation of anotheror others of the display devices. The display device which detects thelocations/orientations of the other devices either provides relativedistance/direction data to the other devices, or to a central computingunit.

In some embodiments, the detecting a location and/or orientation of thedisplay devices is optionally performed by a central tracking unit, e.g.such as a tracking camera and a tracking processor, or a Kinect camera.

In some embodiments a central computing unit calculates the pixelsettings for SLMs of each of the CGH image displays.

In some embodiments each of the CGH image displays calculates the SLMpixel setting for itself, based partly on receiving a location of thereal body relative to the CGH image display.

The above-mentioned embodiments span many combinations. One suchcombination is described with reference to FIG. 3 . A person skilled inthe art can understand many additional combinations based on thedescription of the example embodiment of FIG. 3 .

Reference is now made to FIG. 3 , which is a simplified illustration oftwo HMDs displaying a same holographic image of a first object behind orwithin a visually obstructing second object, according to an exampleembodiment of the invention.

FIG. 3 depicts a holographic image 210 of the first object, for exampleinternal organs, for example bones and lung, behind or within thevisually obstructing second object, for example the skin of an actualbody 211 of the patient whose bones are displayed.

FIG. 3 depicts two viewers 201 a 201 b each wearing a HMD 202 a 202 b,which displays the holographic image 210 of the bones and lung of apatient.

The viewers 201 a 201 b also see the body 211 of the patient, so see theinternal organs (bones and lung) and the body 211.

In some embodiments, the displays, such as the HMDs 202 a 202 b, aresee-through displays, such that viewers see the real world and the body211 through the display at the same time as seeing the CGH image 210 ofthe internal organs (bones and lung). Such a display is termed anaugmented reality display, since both reality and the CGH image areseen.

In some embodiments, the CGH image 210 of the bones and lung iscorrectly located in space relative to the body 211.

In some embodiments the display systems, the HMDs 202 a 202 b forexample, obtain a first relative location or coordinates and/ororientation of a marking or markings 226 a 226 b on the body 211 of thepatient, relative to the HMDs 202 a 202 b, and also obtain a secondrelative location or coordinates and/or orientations of the markings 226a 226 b relative to a three-dimensional imaging dataset used forproducing the holographic image 210 of the bones and lung of thepatient. Each one of the HMDs 202 a 202 b produce a CGH image 210display of the bones and lung correctly located in space relative to thebody 211, by co-registering the CGH image 210 to the body 211. In suchembodiments the HMDs 202 a 202 b operate independently of each other,yet optionally display the same CGH image at the same location andorientation in space and relative to the body to the above-mentionedappropriate degree of accuracy.

In some embodiments the display systems, the HMDs 202 a 202 b forexample, obtain a first relative location or coordinates and/ororientation of a marking or markings 226 a 226 b on the body 211 of thepatient, relative to one of the HMDs 202 a 202 b, and also obtain asecond relative location and/or orientation of the other one of the HMDs202 a 202 b. Each one of the HMDs 202 a 202 b produce a CGH image 210display of the bones and lung correctly located in space relative to thebody 211.

In some embodiments the relative location or orientation of the otherone of the HMDs 202 a 202 b is optionally obtained by one or moresensors 204 a 204 b 204 c 204 d on the HMDs 202 a 202 b optionallydetecting one or more marking or sources of light on the other one ofthe HMDs 202 a 202 b, e.g. along directions 207 a 207 b 207 c 207 d.

An aspect of some embodiments of the invention involves displaying, byway of a non-limiting example, a CGH image of a body's internal organs,correctly placed within a view of the real body, and a CGH image of anadditional object, for example a surgical tool, inserted wholly orpartially within the real body, correctly placed within the view of thesame real body, to the above-mentioned appropriate degree of accuracy.

An aspect of some embodiments of the invention involves displaying, byway of a non-limiting example, a CGH image of a body's internal organs,placed not aligned with the real body, and providing user interfacecommands for a physician to align the CGH image with the real body. Somenon-limiting example of the user interface commands include:

automatically aligning the CGH image with the real body, potentiallyincluding automatic shifting and/or scaling and/or rotating;

automatically moving an aligned CGH image to a different position, suchas floating above the real body, or in the vicinity of the real body;and

using a user interface control, such as a slider, mouse, wheel,joystick, touchpad, pointer to move the CGH image in three-dimensions,to enlarge or shrink a display of the CGH image, to rotate the CGHimage.

In some embodiments the user interface is optionally implemented bydetecting hand and/or finger motions in a space monitored by the CGHdisplay system, or by eye movements monitored by the CGH display system.

In some embodiments a physician optionally aligns the CGH image with thepatient's body by moving and/or rotating and/or scaling the CGH imagedisplay without handling the patient.

In some embodiments a physician optionally aligns the CGH image with thepatient's body by using depth cues such as eye focus accommodation andeye convergence to determine a distance in a CGH image depth to placethe CGH image relative to the patient's body, optionally aligningvisible parts of the patient's body with corresponding parts in thedisplayed CGH image.

In some embodiments the markings 226 a 226 b optionally have a structurewhich enables a sensor or sensors such as sensor 204 a, or an HMD suchas the HMDs 202 a 202 b to determine orientation of the marking inspace. By way of a non-limiting example, FIG. 3 depicts markings with ashape of a letter “F”, which shape enable differentiating between up,down, left and right in an image. In some embodiments the shape of themarking, such as the shape of letter “F”, to indicate a slope toward oraway from a viewer by detecting a convergence of parallel sides of theshape. In some embodiments the shape of a letter “R” is used todetermine the position of an additional plane. In some embodiments, amarker is used which has geometric features that enable determining adirection of the marker, such as differentiating between up, down, leftand right in an image.

In some embodiments enough markings, such as the markings 226 a 226 b,are included on the body so as to enable shifting along one, two orthree (X-Y-Z) spatial axes, rotating around one, two or three axes, andscaling of a displayed image to align with the body.

In some embodiments some of the markings may be embedded within thebody, as described elsewhere herein, and the relative distance and/ordirection of at least some of such markings is optionally known relativeto marking on the body. A dataset for displaying the CGH image, acquiredby some imaging modality, includes at least some of the internalmarkings. The CGH image display system optionally aligns the CGH imageto the body based on detecting the markings on the body and displayingthe CGH image at a correct displacement from the internal markings andthe above-mentioned relative distance and/or direction.

Reference is now made to FIGS. 4A and 4B, which are simplifiedillustrations of a HMD displaying a holographic image of a first objectbehind or within a visually obstructing second object, as well as anadditional object at least partially behind or within the visuallyobstructing second object, according to an example embodiment of theinvention.

FIG. 4A depicts the holographic image 210 of the first object, forexample internal organs, for example a liver, within the visuallyobstructing second object, for example the skin 211 of an actual body ofthe patient whose liver is displayed, and the additional object, forexample a surgical tool 209, for example a syringe which includes aneedle 214.

FIG. 4A depicts a viewer 201 wearing a HMD 202, which displays theholographic image 210 of the liver of a patient.

The viewer 201 also sees the skin 211 of the body of the patient, so theviewer 201 sees both the internal organs (liver) and the skin 211.

FIG. 4B depicts the holographic image 210 of the liver, within thevisually obstructing skin 211, and the surgical tool 209, the syringe,and the needle 214. However, FIG. 4B depicts the needle 214 partiallybehind the skin 211.

The viewer 201 also sees the needle 214, partly outside the skin 211, asa real object, and partly below/behind the skin 211, that is within thebody, as a CGH image of the needle 214, or at least of the portion ofthe needle 214, which is within the body.

In some embodiments the HMD 202 detects a location and/or orientation ofthe surgical tool 209, and/or of the needle 214, and displays the needle214 as a CGH image of the needle 214, co-registered with, or alignedwith, the actual needle 214. The viewer 201 thus sees the entire lengthof the needle 214. The HMD 202 displays a portion of the needle 214which is outside the skin coinciding with a view of the portion of theneedle 214 which is outside the skin, and the HMD 202 displays a portionof the needle 214 which is within the body at the location which thewithin-the-body portion of the needle 214 actually exists. The viewer201 sees both the skin 211 as a real view, and the entire length of theneedle 214, partly as a CGH image and partly as a real view. The HMD 202calculates and displays an extrapolated or calculated image of thehidden portion of the needle 214.

Since a CGH image provides actual depth cues such as eye focusaccommodation and also eye convergence, the CGH image of the needle 214is seen on both sides of the skin 211 as an augmented reality image.

In some embodiments a portion of the tool or needle 214 which is hiddento a naked eye, and is below the skin 211 is sensed by an imaging systemsuch as an X-ray imaging system or an ultrasound imaging system. Theportion of the needle 214 which is hidden is displayed as a CGH image,optionally correctly aligned with a portion of the needle which isvisible to a naked eye.

In some embodiments markers on the visible portion of the too or needle214 are detected by the HMD 202 and the location of the hidden portionof the needle 214 is calculated based on knowing the shape of the toolor needle 214 and on detecting the location of the markers.

In some embodiments the HMD 202 detects a location and/or orientation ofthe surgical tool 209, and/or of the needle 214, and displays just theportion of the needle 214 which is inside the body as a CGH image of theneedle 214. The HMD 202 optionally detects a shape of the needle 214 andproduces just the hidden portion of the needle 214.

In some embodiments the HMD 202 detects a location and/or orientation ofthe surgical tool 209, and/or of the needle 214 using an imaging sensor(not shown) attached to or part of the HMD 202.

In some embodiments the HMD 202 detects the location and/or orientationof the surgical tool 209, and/or of the needle 214 using imageprocessing of an image captured by the imaging sensor attached to orpart of the HMD 202. In some embodiments the image processing detectsand locates the needle 214 based on detecting a shape of the surgicaltool 209, and/or of the needle 214. In some embodiments the imageprocessing detects and locates the needle 214 based on detecting one ormore marker(s) attached to or marked on the surgical tool 209, and/orthe needle 214.

In some embodiments, the HMD 202 also displays a skin 206 c of thepatient. In some embodiments, the skin 206 c is displayed based on datafrom a three-dimensional (3D) imaging dataset used for producing theholographic image 210 of the first object, for example the liver of thepatient.

In some embodiments an additional optical tracking system and/orelectromagnetic tracking system detects the location and/or orientationof the surgical tool 209, and/or of the needle 214. In some embodimentsimage processing of images from the optical tracking system detects andlocates the needle 214 based on detecting a shape of the surgical tool209, and/or of the needle 214. In some embodiments the image processingand/or the electromagnetic tracking system detects and locates theneedle 214 based on detecting one or more marker(s) attached to ormarked on the surgical tool 209, and/or the needle 214.

In some embodiments the display system, the HMD 202 for example, obtainsa first relative location or coordinates and/or orientation of a markingor markings 206 a 206 b on the body 211 of the patient, relative to theHMD 202, and also obtains a third relative location or coordinatesand/or orientations of one or more marking(s) 212 a (212 b) relative,for example, to the HMD 202.

In some embodiments, the HMD 202 displays a holographic image of thetool 209 so that the marking(s) 212 a (212 b) location in theholographic image of the tool 209 coincides, also termed hereinco-registers, with the marking(s) 212 a (212 b) location in the realworld. Such co-registration potentially displays the tool 209 in thecorrect location relative to the body 211.

In some embodiments the HMD 202 optionally has one or more sensor(s) 204which can detect and locate the marking(s) 212 a (212 b). The sensor 204optionally measures distance and/or angle toward the marking(s) 212 a(212 b) on the tool 209, for example along a line between two markings212 a 212 b.

In some embodiments an orientation is optionally determined by detectingan optionally asymmetric shape of the markers.

The sensor 204 may optionally be any one of the sensors describedherein.

The marking(s) 212 a (212 b) may optionally be any of the markingsdescribed herein.

In some embodiments, tracking a tool's location and/or orientation inspace relative to the HMD 202 is optionally performed by an externalsystem tracking the tool 209 and/or the HMD 202.

Reference is now made to FIG. 4C, which is a simplified illustration ofa HMD displaying a holographic image of a first object, obtained andregistered in real time, behind or within a visually obstructing secondobject, as well as an additional object at least partially behind orwithin the visually obstructing second object, also obtained andregistered in real time, and one or more additional guide lines,according to an example embodiment of the invention.

FIG. 4C is similar to FIG. 4B, and displays a similar scenario, and alsodepicts optional guide lines 220, which are optionally produced toillustrate a planned path for the needle 214.

In some embodiments the guide lines 220 are optionally co-registeredwith the holographic image 210 of the first object, for example theliver.

In some embodiments the guide lines 220 are optionally displayed orun-displayed according to a user input.

In some embodiments the guide lines 220 are optionally automaticallydisplayed in space up until the skin of the body 211, and/or up untilthe liver (210), and/or from the skin of the body 211 up until the liver(210).

Co-Registration

An aspect of some embodiments of the invention involves co-registering,or aligning, one or more CGH images with a real-world scene.

Co-registering a CGH image with a real world scene includes aligning theCGH image to a real world scene including location and/or orientationand/or sizing and/or scaling to the real world scene.

In terms of depth perception, a CGH image which is co-registered with areal object provides eye convergence and eye focus depth cues forviewing the CGH image which are the same as viewing the real object.

In some embodiments a first CGH image is co-registered to a truelocation of a body used for obtaining data for producing the first CGHimage, and a second CGH image, optionally of an element which isinserted to the body, such as stent, can be viewed as part of a combinedCGH image including both the first CGH image and the second CGH image.Such combination may be used, by way of a non-limiting example, tomeasure compatibility of the element with the body. The second CGH imagecan optionally be moved, rotated and rescaled with respect to the firstCGH image.

In some embodiments, the display system which produces the CGH imagelocates the CGH image so that points/locations in the CGH image coincidewith associated points/locations in the real world scene. For example,the points which the CGH image display causes to coincide may optionallybe points which the CGH image display receives as specific marker pointsin a three-dimensional dataset for producing the CGH image, and whichare also detected by the CGH display in the real world. For example, thepoints which the CGH image display causes to coincide may optionally bepoints which the CGH image display detects by image processing thethree-dimensional dataset to detect the specific marker points in thethree-dimensional dataset, and which are also detected by the CGHdisplay in the real world. The process of detection is described in moredetail below, in sections named “Markers” and “Detecting markerlocation(s)”.

Causing the marker points to coincide may optionally involve one or moreof the following actions: shifting the CGH image to a specificthree-dimensional location in real space, rotating the CGH image, andscaling, or resizing the CGH image.

Co-registering two CGH images with a real world scene, for example a CGHimage of hidden objects and a CGH image of a partially hidden tool,includes causing the two CGH images to be located/shifted and/ororiented and/or sized/scaled the same as the real world scene.

In some embodiments the three-dimensional dataset for producing the CGHimage of internal organs may have been acquired prior to activating theCGH image display system. For example, the CGH system may display a 3Dmedical CGH image from a medical CT scan performed minutes, hours, days,and weeks prior to the displaying.

In some embodiments the three-dimensional dataset for producing the CGHimage may be acquired concurrently, optionally within seconds or less,to activating the CGH image display system. For example, the CGH systemmay display a 3D medical CGH image from an Ultrasound device being usedto scan a patient, and the 3D medical CGH image may be displayed at itscorrect location within the patient's body.

In some embodiments a marker that is visible to the CGH image displaysystem may optionally be connected to the ultrasound sensor's handlesuch that the data acquired by the ultrasound sensor has a knowndistance from the marker. By detecting the marker position andorientation it is possible to display a CGH image produced from theultrasound imaging tool co-registered with the real location of thedisplayed data.

In some embodiments the three-dimensional dataset for producing the CGHimage of a tool may have been acquired prior to activating the CGH imagedisplay system. For example, the CGH system may display a 3D medical CGHimage of a tool from a library of tool images, where the library may beproduced by some form of 3D image acquisition minutes, hours, days, andweeks prior to the displaying, or even provided by the toolmanufacturer.

Registration Chaining

An aspect of some embodiments of the invention involves chainingregistration of several bodies, each with registration locations locatedthereon.

In some embodiments a pair of three-dimensional (3D) image data sets forco-registering is provided, each one of the pair including one or moreregistration locations detectable by a display system. By way of anon-limiting example, a first 3D data set for producing a CGH image of afirst object, including coordinates of one or more registrationlocations in the 3D data set, and the first object from which the 3Ddata set was acquired, including coordinates of the one or moreregistration locations in the first object, for example as detected by aCGH image display system.

In some embodiments an additional source for co-registering is provided,in addition to the above-mentioned two sources, the additional sourceincluding one or more registration locations also detectable, either bythe display system, or by the imaging modality used to acquire the first3D data set. By way of a non-limiting example, the additional source maybe a second 3D data set for producing a second CGH image of a secondobject, including coordinates of one or more registration locations inthe second 3D data set, and relative positions of coordinates of the oneor more registration locations in the first 3D object, or in the firstobject, relative to the registration locations in the second object.

In some embodiments the co-registration of any number of CGH imagesand/or objects may be performed, using registration locations indatasets for producing the CGH images and/or in objects, as long asrelative positions of the registration locations are known. In someembodiments the registration locations are the same in pairs or more ofthe data sets and/or the objects, so the relative positions in space,when the CGH images and/or the objects are aligned, are the same.

Registration by User Interface (UI)

An aspect of some embodiments of the invention involves a viewerproviding commands to a user interface, for the CGH image of a firstobject to shift the CGH image of the first object in space in one, twoor three-dimensions, and/or to rotate the CGH image in space, and/or toscale (shrink or enlarge) so as to register the CGH image of the objectbehind or within another, second object.

In some embodiments the viewer may perform the registration by eye,optionally using only vision cues. It is noted that a CGH image providesvisual depth cues including both eye focus accommodation and eyeconvergence, so such registration by eye feels natural, moving a CGHimage to its correct location.

In some embodiments the viewer may perform the registration by aligningmarkers, as described elsewhere herein. Aligning markers by sight may betermed registering by using registration aids.

In some embodiments the viewer performs the registration by moving theCGH image in the depth direction to achieve identical eye focusaccommodation and convergence to a registration location in the CGHimage as to a corresponding location in the real object.

Automatic Registration

An aspect of some embodiments of the invention involves a CGH displayautomatically aligning the CGH image of a first object to shift the CGHimage of the first object in space in one, two or three-dimensions,and/or to rotate the CGH image in space, and/or to scale (shrink orenlarge) so as to register the CGH image of the object behind or withinanother, second object.

In some embodiments the automatic alignment may optionally be offeatures in the CGH image of the first object and features in the secondobject, the features being detectable by image analysis.

In some embodiments the automatic alignment may optionally be ofmarkers, as described elsewhere herein. Aligning markers may be termedregistering by using registration aids.

Markers

Markers, also termed markings, used for aligning images of 3D datasetswith 3D objects in the real world, may be of various types.

The term “marker” in all its grammatical forms is used throughout thepresent specification and claims interchangeably with the terms“reference marker”, “marking” and “reference marking” and theircorresponding grammatical forms.

In some embodiments, markers are attached to a real world object such asa patient's body. Such markers are optionally visible/detectable bysensors in the 3D CGH image display system and also by a 3D medicalimaging acquisition system such as a CT system. Such markers mayoptionally be metal markers, which are detectable by a CT system,optionally colored a color which shows up against the patient's body,and/or optionally textured with a texture which shows up against thepatient's body.

In some embodiments, where a marker should not be made of magneticmaterial, such as some markers intended for a specific imaging modalitysuch as Magnetic Resonance Imaging (MRI), a material which showscontrast with a patient's body in the specific imaging modality isoptionally used.

In some embodiments, a marker is drawn at a specific location on apatient's body or object, optionally using a color which shows contrastwith the patient's body or object.

In some embodiments, a marker comprises a light source such as a LEDlight source.

In some embodiments, a marker comprises a shape that indicatesorientation, or direction. Some non-limiting examples include an arrowshape, or a shape that is different from a mirror image of the sameshape, such as the letters “F”, and “R”.

In some embodiments, a marker which serves to co-register two 3Ddatasets is optionally detectable by both imaging modalities used toproduce the 3D datasets.

In some embodiments, a marker which serves to co-register a CGH image ofa 3D dataset and a real object is optionally detectable both by theimaging modality used to produce the 3D dataset and by the CGH imagedisplay system.

In some embodiments a marker may be attached to an outside of abody/object.

In some embodiments a marker may be inserted into a body/object, and/orimplanted in a body/object.

In order to determine a three-dimensional orientation of athree-dimensional body or object in a coordinate system it is enough toknow locations of three points in the object. It is noted that knowinglocations of more than three points may optionally provide redundantinformation, which in some embodiments may be used to increase accuracyof registration.

In some embodiments a single marker is used per body/object, and thethree points are located on the single marker. For example, a markerwith a shape such as the letter “F” or other shape as mentioned aboveprovides optionally three points, such as a base of the leg of the “F”and the two ends of the two arms of the “F”.

In some embodiments two markers are used per body/object, and three ormore points are located distributed on the two markers.

In some embodiments three markers are used per body/object, and three ormore points are located on the three markers. In some embodiments, onepoint is located per marker. In some embodiments the markers areoptionally small, and detecting locations of the markers is used asdetecting point locations.

In some embodiments the markers are located at various depths in a bodyor object. By way of some non-limiting examples the markers may beplaced in one plane, that is, for example three or four or more markersmay be placed on one geometric plane. By way of some non-limitingexamples the markers may be placed in different planes, that is, forexample four or more markers may be placed on more than one geometricplane.

Additional example embodiments include markers such as a micro needle,optionally of less than 5, 10, 50, 100, 500, 1,000, 2,000, 5,000-microndiameter; a micro clip, for example of 2 mm diameter.

In some embodiments the markers are active, that is, the markers emitsignals for sensing by a detector. Some non-limiting example of activemarkers include Bio sense by Webster Inc.; a Given Imaging capsule; alight emitting marker, e.g. a LED light.

In some embodiments the markers are passive, that is, the markers do notactively emit signals. Some non-limiting example of passive markersinclude a magnetic marker; paint; natural body markers; needles attachedto or stuck in the patient; ink, and so on.

In some embodiments registration may optionally be performed on a movingbody, optionally by detecting a specific time in a repetitive movement,such as a specific point in a breathing cycle.

In some embodiments the markers are natural features that are detectableby the CGH projection unit and are part of the body, such as the contourof an organ, the center of an eye, a bone segment, the umbilicus, theend point of a tool.

The term marker “on a body” or “on a body organ”, in all its grammaticalforms, is used throughout the present specification and claimsinterchangeably with the term marker “in a body” or “in a body organ”and their corresponding grammatical forms. A person skilled in the artwill discern when a marker is “on” a body or body organ, that isapproximately at a surface of the body or body organ, and when a markeris “in” a body or body organ, that is within the body or body organ.

Detecting Marker Location(s)

Various sensors or imaging technologies may be used in a CGH displaysystem to detect markers or to detect body/object locations to use forco-registration.

Some example embodiments of sensors include cameras, optionally used inconjunction with image processing.

Some example embodiments of sensors include ultrasound imaging, whichcan optionally be used even real-time, during a physician performing amedical procedure, surgery, or diagnostics, to detect a marker and use alocation of the marker to register an organ or a tool marked by themarker to a patient's body or limb.

Some example embodiments of sensors include ultrasound imaging, whichcan optionally be used even real-time, during a physician performing amedical procedure, surgery, or diagnostics, to detect a marker and use alocation of the marker to register an organ or a tool marked by themarker to a patient's body or limb.

Some example embodiments of sensors include TransesophagealEchocardiogram (TEE) imaging, which can optionally be used evenreal-time, during a physician performing a medical procedure, surgery,or diagnostics, to detect a marker and use a location of the marker toregister an organ or a tool marked by the marker to a patient's body orlimb.

In some embodiments a Transesophageal Echocardiogram (TEE) imaginginstrument which is typically inserted into a patient body, isoptionally imaged by another imaging modality, such as x-ray or CT,which detects a location in space of the TEE instrument, and a CGH imageproduced by a TEE imaging system is optionally aligned based on thelocation in space of the TEE instrument.

An example embodiment is now described which is taken from a differentfield than medicine—for example using see-through vision in a context of“seeing through” walls of a building, to view images of elements such aspipes or wiring appear in their correct 3D location within or behindwalls, or “seeing through” the ground to view images of elements such aspipes appear in their correct 3D location underneath the ground.

In some embodiments, a camera and/or a thermal camera may optionally be,pointed at a wall and be used to identify cold and/or hot spotsoptionally used as markers. The cold/hot spots are optionally thermalelements inserted in the wall, and optionally brought to a temperaturedifferent from the rest of the wall, for example an electrical resistorbeing heated by current, such that its location is identified.

Markers are not necessarily stationary. By way of a non-limitingexample, in some embodiments a marker is optionally inserted into aflowing medium and can be used to track flow speed, openings in the flowchannel and or blockages. The marker is optionally a solid object ofsizable volume, a powder, a fluid. In some embodiments a moving markeris optionally used in conjunction with one or more stationary markers.In some embodiments, the CGH image is aligned with a real body/object,and the non-stationary marker(s) are optionally detected by a 3D imageacquisition system and the location of the non-stationary marker(s) isoptionally calculated and optionally displayed.

An aspect of some embodiments of the invention relates to holographicimaging, for example by a holographic head-mounted display, whichenables presentation of an image produced from a 3D data set, inabsolute coordinates. Such presentation optionally includes displayingthe 3D data with natural depth cues used by viewers, including differentfocii for different points in the image at different distances from aviewer, within a unified coordinate system. Viewers of such imagesperform eye convergence and eye focus to view different points in a CGHimage which appear at different distances from the viewers.

In some embodiments, such an image may be calculated and projected inreal time, optionally even at video rate, at a distance that is a hand'sreach from the viewer. The image potentially appears floating in space,allowing a viewer to optionally insert his/her hand or a tool into theimage space, and interact with the image just as he/she would interactwith a real 3D object. The viewer can mark a location on or in theimage, move the image by grabbing it and rotating it as he/she wishes.More complex manipulations can also be performed. The viewer can changehis/her position while the image stays fixed in space as a real objectwould.

In some embodiments, for multi-user interaction, when a person touchespart of the image all the other viewers see that part with the fingertouching just as a real object would appear.

In some embodiments, for a holographic Head-Mounted Display (HMD) or aholographic head set display, the Computer Generated Hologram (CGH)image is an imaginary holographic image, that is, the light seen by theeye, does not originate from the CGH image directly to the eye, but isredirected by an optical element. Such an attribute can optionally beapplied for making the CGH image appear at locations where real worldobjects would not otherwise be visible.

In some embodiments, 3D see-through vision provides the ability to fuseor integrate or overlay a holographic image of an object and a realworld solid object. The holographic image may optionally display 3Dimages at locations where objects are not expected to be visible to thenaked eye, and projected as being transparent and/or having variabledegrees of transparency, opacity or translucence.

Consider a scenario in which a holographic image is derived from 3D dataof the internal aspects of the solid object. By optionally integratingand or overlaying the holographic image with or on the solid object, aviewer has the ability to apparently see through the solid object as ifit were transparent or translucent. Internal aspects of the object,and/or surfaces beyond or underneath a surface of the affected object ormaterial can be seen.

Consider an object, in plain sight, which the viewer can see in 3D, withnormal depth perception. In some embodiments, a 3D hologram of the sameobject, including detailed information of the object, includinginformation that is not visible in plain sight, for example, informationthat is internal to the object, or not in the field of view of theviewer, may optionally be displayed.

In some embodiments, the holographic image is optionally overlaid on thereal object, such that the viewer can see the object, as he wouldnormally view it, while also viewing internal aspects of the object andas well as facets of the object that are not obtainable by naturalsight, e.g. the sides and or back side of the object.

In some embodiments both the holographic image and the real object arevisible to the viewer. In some embodiments the viewer focuses his eyeson the holographic image or the real object. Such focusing is naturallyperformed in human vision, and the holographic image provides full depthcues to human vision, so the viewer may optionally select what to viewin focus, the holographic image or the real object.

In some embodiments, 3D see-through vision provides potentialapplications where 3D image acquisition is used. For example, in amedical application, for example in non-destructive testingapplications, such as non-destructive testing of structures and ofwelds, and for example in seismic studies.

In some embodiments such as nondestructive testing (NDT), imagescollected optionally include 3D data. By way of a non-limiting example,in weld inspection, identifying cracks or flaws typically requires skilland experience in interpreting a 2D display of the 3D data, whichtypically provides a small slice of a flaw. Inspection of a 3D hologrampotentially enables making the interpretation task intuitive, andfurthermore optionally displays the flaw/crack in its correct locationwith respect to the component under test. Cracks are typically of smalldimensions, by way of some non-limiting examples having one of thedimensions of the crack less than 1, 5, 10, 20 mm. Inspection istypically conducted at close range, for example within hand-reach, orless than 25, 50, 75, 100, 150 centimeters away from a viewer. NDTinspection typically potentially benefits from accurate depth cues,specifically eye convergence and focus accommodation.

Coordinate Systems

The terms global positioning system or global coordinate system orabsolute coordinates are used herein interchangeably to mean a unifiedcoordinate system used to display several objects, where some of theobjects in the display may have been captured using different imagingsystems, which used different coordinate systems, and are now unified touse a common system.

Independent acquiring data systems, such as MRI, CT, Ultrasound, X-ray,electroanatomical mapping systems, provide data in a coordinate systemwhich is referenced with respect to the acquiring system. Thus localcoordinates of acquired data are known.

In some embodiments, in order for a display such as a Head MountedDisplay (HMD) to display the acquired data in a real world globalposition, optionally where a real object appears, the local coordinatesystem of the acquired data is optionally translated and/or rotatedand/or rescaled, to co-register with a coordinate system of the HMD.

By way of a non-limiting example, when a viewer is viewing a holographicimage of lumbar vertebra L3 and L4, as the viewer's head moves up theimage of the spine, the viewer is displayed lumbar vertebra L2 and L1,that is, the viewer scans upwards. In contrast, a typical monitor doesnot track a viewer's head motion, and does not track or shift contentsof the display when the display is moved.

2D Presentation of 3D Data

Information derived from 3D image acquisition is typically presented in2D displays. Understanding 3D data displayed on a 2D screen typicallyinvolves manipulation of the displayed data in order to obtain desiredviews. The understanding typically involves interpretation by the userwho mentally connects multiple cross sections to form a mental 3D image.Such interpretation involves significant skill and experience, and isprone to erroneous observations and conclusions in that the 2Dinformation lacks information, even if multiple 2D images are reviewed.

For example, in medical applications, a clinician interprets 2D imagesthat are sometimes displayed on multiple screens or monitors, andmentally visualizes the relationships of the anatomy and how the 2Dimages relate to the real anatomy of the patient. The ability tounderstand the relationship between the image and that of the patient'sreal anatomy is particularly important, for example as it relates toplanning of minimally invasive interventional surgical procedures. Suchtypes of procedures cause the clinician to mentally fuse the 3D image,which s/he mentally formed by viewing 2D cross sections displayed on a2D screen, with the patient's actual anatomy as s/he actually performsthe procedure. Furthermore, in situations that involve multipleclinicians which communicate and work together, it is beneficial thatthe participants have the same mental 3D image.

In medical applications, the growth of image-guided InterventionalRadiology (IR) and interventional cardiology procedures in surgicalsettings has led to an increased reliance on the use of 3D imaging (CT,cone beam CT, MRI, 3DUS, PET, SPECT). An interventionalist desires ahighly accurate image and a full understanding of the spatial anatomy topinpoint the area of interest for diagnostic procedures such as imageguided biopsy. In addition, with the advancement of image guidedtreatment, an interventionalist should have a comprehensiveunderstanding of the anatomical relationship of neighboring vessels andstructures in order to spare healthy tissue and minimize or preventdamage to the adjacent tissue during the procedure. Furthermore, it ispotentially beneficial that such data be available and interpreted in ashort time span so that intervention durations can be reduced to aminimum. Furthermore, it is potentially beneficial to acquire the 3Ddata with minimal acquisition cycles. Minimal acquisition cycles canreduce the overall intervention procedure time, reduce handling of thepatient and reduce radiation exposure.

The 3D see-through vision system potentially provides a clinician withan intuitive understanding of spatial anatomy based on a displayedhologram. The 3D see-through vision system allows the clinician tovisualize a holographic image fully registered to an actual location inor on a patient's body and displayed as-if through the patient's skin.The system can optionally also provide holographic tracking ofintervention tools such as a biopsy needle/energy deliveryneedle/catheter/camera as they are navigated to tissue of interest.

Embodiments of a 3D see-through vision system can be employed in avariety of clinical areas, for a large range of clinical applications,including but not limited to: Volumetric Tissue Biopsy; Biliary Drainageand Stenting; Chemoembolization; Embolization; IrreversibleElectropolation, Infection and Abscess Drainage; Needle Biopsy;Radiofrequency (or other energy source) Ablation; Urinary TractObstruction; Uterine Artery Embolization; Uterine Fibroid Embolization;Vertebroplasty, Dental Implants, Interventional Neurology.

It is noted that using a 3D see-through vision system has a potential toreduce errors in medical procedures and/or diagnoses and/or reduceduration of medical procedures. The 3D see-through vision systemoptionally displays organs in their correct location within a patient'sbody, potentially preventing mistakes such as accidental right/leftsubstitution by a physician which views an image of the organs on amonitor not aligned or registered with the patient's body.

Volumetric Biopsy

In some example embodiments 3D see-through vision potentially enhances3D anatomical understanding for tumor identification and needleplacement when performing a volumetric biopsy procedure. The 3Dsee-through vision potentially enables intuitive understanding of the 3Dinformation, such as that the 3D see-through vision potentially reducesthe time required for identification of tumors and potentially increasesaccuracy of needle placement in tissue to be biopsied. Such animprovement is potentially further augmented when performing minimallyinvasive treatment, by way of a non-limiting example in cases in whichthe treatment target is very close to a major blood vessel with complexanatomy.

Tissue Ablating Energy

Some minimally invasive methods of treatment involve the delivery oftissue ablating energy by means of a needle or series of coupledneedles. Exact positioning of the needle(s) within the target tissues ispresently performed under fluoroscopic guidance on a pre-procedure CT.

An example embodiment of a 3D see-through vision potentially enables preor intra-procedure CT data to be integrated to the patient's actualanatomy of the target organ and/or of a needle, potentially enablingdirect needle advancement to an exact position.

High Intensity Focused Ultrasound Treatment

Mention is made of non-invasive methods based on MR guided highintensity focused ultrasound (HIFU) treatment. An objective ofMR-guidance is to control heat deposition with HIFU within the targetedpathological area, despite the physiological motion of these organs.

In such methods several technological challenges exist. Anatomicallocation of both organs within the thoracic cage make intercostalablation strategies desirable, to preserve therapeutic efficiency, butprevent undesired tissue damage to the ribs and the intercostal muscle.Therapy guidance and energy deposition should preferably be renderedcompatible with continuous physiological motion of the abdomen.

In some embodiments, using the 3D see-through vision, real-time MagneticResonance Imaging (MRI) is optionally displayed and or optionallyregistered to the patient's real anatomy. The clinician is thenpotentially able to visualize the trajectory of the HIFU as it isdelivered through the ribs and intercostal muscle, in real-time, andpotentially able to guide the HIFU energy to its target even duringcontinuous physiological motion of the abdomen.

A Simultaneous View of External and Internal Aspects

In some embodiments, using 3D see-through vision, a viewer is able toview external aspects of an object, as he would normally observe anobject, while also viewing internal aspects of the object,simultaneously, and at a same location.

3D see-through vision potentially enables fusing one or multiple 3Dimages, with real world views of the same object. For example, 3D imagesof the internal aspects of the object as well as 3D representations ofvarious properties of the internal object such as temperature, density,electrical properties.

In some embodiments, using the 3D see-through vision and/or real-timetracking capabilities in a procedure in which an energy deliveringcatheter is inserted into a target organ through the skin, the clinicianis able to see a portion of the catheter that is external to the tissue,to see the point of entry, as he normally does. He is also able to seethe portion of the catheter that is within the tissue, as it is beinginserted into the tissue. Both external and internal views areoptionally visible from the same viewing aspect, so that the cliniciandoes not have to shift his eye or move his head towards a separateimaging display.

Using an unaided eye, when viewing a needle or other tools that areinserted through the skin, a portion of the needle or other tooldisappears from view once the portion passes through the skin. In someembodiments, using the 3D see-through vision, the clinician is able tosee the portion of the needle or other tool that is external to the skinwhile also seeing the portion of the needle or other tool that has beeninserted through the skin.

In some embodiments, when treating an organ, such as in open surgery,the 3D see-through vision potentially enables viewing beyond an outerlayer of the organ, similarly to viewing beyond the skin as describedabove.

Another minimally invasive method for treating tumors is by low doseradiation therapy delivered by brachytherapy. Computed tomography(CT)-guided brachytherapy is presently used to treat primary andmetastatic cancer (e.g. liver, breast, prostate cancer). Brachytherapyinvolves precise placement of short-range radiation-sources(radioisotopes) directly at a site of a tumor by means of a needle(s).As with the procedures described above, in some embodiments, byemploying the 3D see-through vision technology during such theprocedure, the clinician is potentially able to visualize the needle asit is being inserted into the tissue, as opposed to the situation todaywhere the needle “disappears” from unaided view once it passes throughthe skin.

In the above examples, procedures conducted while using embodiments of3D see-through vision are potentially performed in less time and withbetter accuracy, potentially resulting in less radiation and/or bettertumor coverage, where applicable, than similar procedures conductedwithout the 3D see-through vision.

In some embodiments, potential added value of the use of the 3Dsee-through vision is applicable to the above described scenarios and tomany other minimally invasive needle/catheter guided procedures such as,but not limited to, Volumetric Tissue Biopsy; Biliary Drainage andStenting; Chemoembolization; Embolization; Irreversible Electropolation,Infection and Abscess Drainage; Needle Biopsy; Radiofrequency (or otherenergy source) Ablation; Urinary Tract Obstruction; Uterine ArteryEmbolization; Uterine Fibroid Embolization; Vertebroplasty, DentalImplants, Interventional Neurology.

Changing a Viewing Mode of a Holographic Image

In some embodiments, using 3D see-through vision, a holographic image ofa patient's anatomy is superimposed onto the patient's actual anatomy,to guide a clinician intra-procedurally. The clinician can optionallychange, optionally ad hoc, a viewing mode of the holographic image, suchthat the image is temporally disconnected from the real world object.For example, the holographic image may optionally be disconnected fromthe patient's actual anatomy to a position that is viewed as floatingabove the patient. In the floating above the patient position, theclinician can optionally interactively manipulate the holographic imageto enhance spatial understanding of the patient's anatomy image andinter-relationship of anatomical elements. In such a mode, the surgeoncan rotate and or slice the holographic image. Furthermore, switchingbetween modes can potentially provide the surgeon with real time viewsof the location of a surgical tool from different viewing angles withrespect to the internal aspects of the organ being treated.

Controlling the holographic image viewing mode, be it connected ordisconnected from the real world object, can optionally be performed bythe viewer, optionally single handed. Controlling the holographic imageviewing mode can optionally be achieved by a software menu visible in aviewing volume adjacent to the patient, optionally using visual cues,such as hand gestures, with voice commands, optionally using eyetracking technology, and similar means known in the art.

Multiple Holographic Images

In some embodiments, a 3D see-through vision system optionally providestwo holographic images derived from a same 3D image data set. Oneholographic image is registered to and appears at a location within thereal world object, and a second holographic image is floating in space,e.g. floating in the air above a patient. Both images are optionallyupdated in real time, optionally simultaneously.

In some embodiments the two holographic images are provided side byside, or in close proximity to each other. A surgeon can optionallychoose to view either one of the images or both.

In some embodiments the images are completely or partially independent,such that the surgeon can choose to manipulate, rotate, cut and/or sliceone of the holographic images without affecting the other image(s).

In some embodiments the 3D see-through vision system optionally providesmultiple holographic images derived from the same 3D image data set. Allof the images are optionally updated in real time, optionallysimultaneously. At least one, or more, holographic image is notdisplayed as within the real world object. Other holographic images areoptionally displayed in different settings or locations.

In some example embodiments all of the holographic images share datasuch that if one viewer touches a holographic image in a particularspot, the location touched is optionally seen, optionally highlighted orotherwise marked, in some other, or even all of the other holographicimages. Similarly, if one viewer manipulates, e.g. rotates or slices aholographic image, the manipulation is optionally performed on someother or even all of the holographic images.

In some embodiments, the holographic images do not share data, such thatif one viewer touches one holographic image in a particular location,the location touched is optionally not seen as touched in the otherholographic images. Similarly, if one of the viewers manipulate, e.g.rotates or slices a holographic image, the manipulation will not beperformed on some or all of the holographic images.

In some embodiments, viewers or a system administrator can define aprotocol governing sharing of data between multiple holographic images.In an example, the protocol may be fixed, dynamic and/or configurableand modified over time.

In some embodiments, an ability to share or not to share information isoptionally used in a multi viewer setting, such as in a complex medicalprocedure with multiple clinicians. The ability to share or not shareinformation is potentially valuable in training and educationalsettings, for documentation purposes, etc.

In some embodiments, a second holographic image is presented with a timedelay with respect to a first holographic image. The time delay mayoptionally be fractions of a second, seconds, minutes or longer. Asharing of information between the holographic images, in onenon-limiting example, is optionally from the first holographic image tothe second holographic image. In a second non-limiting example there isno shared information, that is, the second holographic image optionallydoes not display manipulations conducted on the first holographic image.

Multiple Holographic Images from Multiple Acquisition Systems

In some embodiments 3D image fusion of multi-modal acquisitions mayoptionally be used to further assist in pre-planning andintra-procedural guidance of interventional procedures. The use of fusedimages may optionally be displayed as holographic images as a 3Dsee-through vision holographic image or as a free floating image abovethe patient.

In some example embodiments a 3D see-through vision system optionallyprovides an ability to integrate multiple images obtained from variousimage acquisition technologies. For example, consider an object viewedin plain sight, on which an accurate hologram image, optionally obtainedstatically, e.g. with a CT, is overlaid. Further, consider overlaying anadditional image obtained by a real time imaging technology such asultrasound. The real time imaging, e.g. ultrasound, may even have lessresolution and accuracy then the imaging obtained from the staticimaging, e.g. CT.

A potential benefit of the overlaying of multiple images is tooptionally provide an improved resolution for non-moving elements of theimaged object, and another potential benefit is to optionally improveregistration. The real time image can optionally provide dynamicinformation of changes over time, due for example to inspiration andexpiration. The real time images can also optionally provide informationabout dynamic systems such as a flow regime under various conditions.

Consider a heart valve being viewed. The static analysis can optionallyprovide an accurate positioning of various registration points, whilethe real time image acquisition can optionally provide information ofheart valve dynamics and blood flow in its vicinity.

In some embodiments, two (or more) 3D data acquisition systems areoptionally deployed in parallel. A first 3D data acquisition system is,by way of a non-limiting example, an invasive system, such as anultrasonic catheter, focused on an organ being viewed, e.g. the insideaspects of an artery. A second 3D data acquisition system is, by way ofanother non-limiting example, an external 3D data acquisition system,such as a CT, focused on a location and orientation of the first 3D dataacquisition system with respect to the viewed artery. In such a manner,a clinician can optionally verify that the first 3D data acquisitionsystem is in a correct place and a correct orientation, and/oroptionally that the data acquired from the first 3D data acquisitionsystem is the data sought.

In some embodiments a holographic image produced based on a pre-acquiredCT, of the brain for example, is optionally aligned with anintra-operative microscope image, optionally providing anintra-operative magnified view of at least part of a total holographicimage.

In some embodiments the alignment of the magnified image to a real bodyand/or object is optionally based on placing a single point of themagnified image at a same location as a single point of the realbody/object.

In some embodiments the alignment of the magnified image to a real bodyand/or object is optionally based on placing a line, optionallycorresponding to a direction from a viewer to a point of interest suchas a center of a field of view, at a same location as a correspondingline of the real body/object.

In some embodiments the magnified image is not aligned to the real bodyand/or object, but optionally displayed in a same area as the realbody/object, potentially enabling a viewer, such as a physician, not todivert his view from the direction of the real body/object.

Depth Perception

Holographic imaging provides holographic images which cause a viewer toperform both eye focus accommodation and eye convergence dependent ondistance of a viewed object or location in or on an object.

Stereopsis tricks a viewer's mind in order to create a perception ofdepth, as in is done in stereoscopic movies or stereoscopic photos.Holographic images retain all of the real world depth cues, andspecifically eye focus accommodation and convergence.

Eye focus accommodation and convergence are considered significant cuesin vision, in that they provide distance information. Eye focusaccommodation is effective at close ranges, typically from 5 cm to 2 or3 meters, and convergence is effective at ranges of up to 10 meters.Both of these ranges are potentially relevant to a typical indoorenvironment, and to images such as human anatomy from medical imaging.

Registration

In some example embodiments a physician may optionally be looking at apatient in an operating theatre or procedure room, and performing aminimally-invasive procedure, optionally a procedure in which the organthat is treated is not seen directly by the physician and the guiding isperformed by imaging of 3D acquired data.

In some embodiments, the physician is displayed the surgical tools andthe organ that are inside the patient, as if the procedure was opensurgery. In-order to do so, a CGH image is optionally co-registered witha location of the patient's anatomy in the real world, in a globalcoordinate system.

In some embodiments a co-registration of the CGH image with a locationof a patient's anatomy is optionally achieved using markers that aredetectable by a 3D data acquisition system and optionally by thehologram projection unit.

In some embodiments a co-registration of the CGH image with a locationof a patient's anatomy is optionally achieved using markers whosepositions/coordinates are fed into a 3D data acquisition system and/orinto the hologram projection unit.

In the scenario described above, there are coordinate systems which areoptionally co-registered, including, by way of some non-limitingexamples:

1. G [Global]/L [Local] PS—there are tools and systems in the operatingroom whose position in space is defined by their coordinates in a globaland/or a local positioning system.

2. Components of a holographic image projection unit.

-   -   a. Camera or sensors that detect marker position.    -   b. A Head Mounted Display.

3. A surgical tool—whose local coordinate system may optionally bedefined with markers

4. 3D data acquisition system(s), whose data is optionally integratedinto a holographic image.

Registration Markers

In some embodiments co-registration of a CGH image with a location of apatient's anatomy is optionally achieved with markers that aredetectable by a 3D image data acquisition system and/or by the hologramprojection unit.

In some embodiments co-registration of a CGH image with a location of apatient's anatomy is optionally achieved with markers whose position isoptionally fed into the 3D see-through vision system.

In some embodiments the relative position and orientation of one markerwith respect to the other is input into the CGH image display system. Insome embodiments a relative position/orientation of the markers isdetected by an another, additional, data acquisition (image modality)system.

In some embodiments markers are placed on or within a body of a patient.In some embodiments, markers are attached to organs or parts of theskeletal system. In some embodiments, markers are placed on a surgicaltool or equipment, such as on needles, laparoscopic tools, catheters,tubes and wires.

In some embodiments a hologram projection unit includes a sensor (orsensors) that can sense a position in space and/or an orientation of amarker or multiple markers. In some cases, the sensor can track a movingmarker in real time, e.g. at a video rate of 60 Frames Per Second (FPS)or higher than 24, 30, 48, 50, 72, 100 FPS. For example, a camera, avideo camera, a High-definition video, an IR camera, a G/LPS receiver.

In some embodiments the marker is optionally detectable by the 3D dataacquisition system as well as by the hologram projection unit. Forexample, to be detectable in an X-ray, CT, US or MRI system.

In some embodiments the marker is optionally, by way of a non-limitingexample: a surgical site marker applied with a pen on skin, fastenersand bone screws, cranial screws, head bolts, MRI marker, clips used inthe liver and soft tissue markers.

In some embodiments a marker can optionally be a point/dot marker or amarker that includes a pattern, or shape, e.g. semi-spherical, orcone-like, positioned on or connected to the skin.

In some embodiments a marker can optionally be a shape which indicateslocation as well as direction, such a shape of a letter “L”, “R”, “F”,and so on.

In some embodiments a marker may optionally have metal or liquidcomponents. In some embodiments, the marker may optionally have activecharacteristics such as emission of RF, heat, magnetic, light or otherradiation. In some embodiments, the marker may optionally have aspecific ID as in RFID or a visual ID placed on markers.

In some embodiments the marker may optionally encompasses an area, forexample, surrounding a portion or all of an exterior surface of acatheter, along the catheter's length, or a line marker on a rigid toolsuch as a needle or a bone.

In some embodiments markers, such as powder or liquid, are optionallyintroduced into fluid systems, such as veins and or arteries.

In some embodiments, the marker may optionally be an object such as aviewer's hand, or anatomical features or portions of a patient's organbeing viewed.

In some embodiments a marker's coordinates, in a local or a globalcoordinate system, may optionally be fed in to the system and thenpresented in the hologram.

Registration in Real Time

Consider medical procedures that include surgical tools such as needles,fibers, endoscopes, catheters, energy delivering systems, and implantsthat are inserted into the body.

In some embodiments the tools include markers or features detectable bya real time acquisition system, or the position of the tools or markerscan be introduced in to the real time acquisition system, that is thesystem that provides the 3D image data to the CGH image display system,and that the coordinates of the tool's shape with respect to the markersis known. When the tool is in image space, the space where the CGH imageappears, the tool image is optionally projected at the same location asthe real tool which is inside the body. That is, the tool image appearsin the body as if the physician sees the tool in an open surgery.

In some embodiments, for example when a tool is rigid, such as a needle,one marker is sufficient. In some embodiments two markers aresufficient.

In some embodiments, for example when a tool is flexible and/or maychange shape and or orientation during a medical procedure, multiplemarkers may optionally be deployed. In some embodiments involvingflexible tools a location of each part of the tool's system is known,optionally in a local coordinate system. In such a case it is sufficientfor one or two markers to co-register a specific part of the tool'ssystem in the coordinate system of the CGH image, thereby co-registeringat least some of the tool's other parts in the CGH image coordinatesystem.

In some embodiments, the markers are optionally placed before or afterjoints of the tool mechanism. In some embodiments, the markers areoptionally distributed at fixed or non-fixed distances along the tool.In some embodiments, the whole tool or portions of the tool areoptionally marked and detectable by a real time acquisition system.

In some embodiments, different types of marking are optionally used onthe same tool. Different types of marking used on the same toolpotentially enable detection of tool orientation.

The marker can optionally be flexible, such as, for example, a catheterin a blood vessel which has sensors, and their positions can optionallybe monitored in real time and directly fed in to the acquisition system.

In some embodiments a real time data acquisition with a real time dataacquisition tool such as an ultrasound system is used, wherein therelative location of the acquired data with respect to the coordinatesystem of the real time acquisition tool, the ultrasound handle forexample, is known. A relationship between the coordinate system of thereal time acquisition tool and the coordinate system of the CGHprojection system is optionally defined using markers on the handle, andby measuring the location and/or orientation of the real timeacquisition tool relative to the coordinate system of the CGH projectionsystem.

An example embodiment using real time acquisition and registration ofimages is described below with reference to FIG. 4D.

Reference is now made to FIG. 4D, which is a simplified illustration ofa HMD displaying a holographic image of a first object, obtained andregistered in real time, behind or within a visually obstructing secondobject, as well as an additional object at least partially behind orwithin the visually obstructing second object, also obtained andregistered in real time, according to an example embodiment of theinvention.

FIG. 4D depicts a holographic image 410 of the first object, for exampleinternal organs, for example a liver, within a visually obstructingsecond object, for example skin 411 of an actual body of the patientwhose liver is displayed, and an additional object, for example asurgical tool 409, for example a syringe which includes a needle 414.

FIG. 4D depicts a viewer 401 wearing a HMD 402, which displays theholographic image 410 of the liver of a patient. The viewer 401 alsosees the skin 411 of the body of the patient, so sees both the internalorgans (liver) and the skin 411.

FIG. 4D depicts the holographic image 410 of the liver, behind thevisually obstructing skin 411, and the surgical tool 409, the syringe,and the needle 414. FIG. 4D depicts the needle 414 partially behind theskin 411.

In some embodiments the holographic image 410 is optionally displayedbased on a 3D data set obtained in real time, for example by a real timeacquisition system such as an ultrasound system or an X-ray system.

The viewer 401 sees the needle 414, partly outside the skin 411 andpartly below/behind the skin 411, that is within the body, as a CGHimage of the needle 414.

FIG. 4D depicts a tip 421 of the needle 414 producing a dimple 420, orindentation, in the holographic image 410 of the liver. The dimple 420is best depicted in an enlarged view in the upper left portion of FIG.4D.

FIG. 4D depicts an embodiment which potentially provides real timevisual feedback to a surgeon inserting a tool such as a needle,potentially augmenting sensory feedback of the needle's resistanceagainst the liver. In some embodiments the real time visual feedback isprovided even when sensory feedback is not felt, possibly owing to weakresistance of the liver to penetration by the needle 414.

Aligning of the holographic image 410 and the holographic image of theneedle 414 to the body of the patient is optionally performed by imageprocessing 3D data sets for producing the holographic image 410 and theholographic image of the needle 414 and an image acquired of the body ofthe patient by the HMD 402. In some embodiments the image processingdetects natural features of the liver/tool/body which may be aligned. Insome embodiments the image processing detects artificial markers, suchas described elsewhere herein.

In some embodiments the markers, for example optical or electrical oracoustic markers, are optionally connected to the real time dataacquisition tool such that they are detectable by the CGH imagingsystem.

In some embodiments some of the markers may be embedded within the body,as described elsewhere herein, and the relative distance and/ordirection of at least some of such markers is optionally known relativeto markers on the body, viewable and/or detectable by the CGH imagedisplay system. A dataset for displaying the CGH image, acquired by someimaging modality, includes at least some of the internal markers. TheCGH image display system optionally aligns the CGH image to the bodybased on detecting the markers on the body and displaying the CGH imageat a correct displacement from the internal markers and theabove-mentioned relative distance and/or direction.

The markers enable monitoring the acquisition tool's coordinates in theimage space, thus the imaging of the acquired data is mapped to thecoordinate system of the CGH imaging system. Such markers include, byway of some on-limiting examples:

a. Optical markers or fiducial markers, such as dye or stickers withspecific patterns, or patterns embedded on or into the tool or LEDs,connected to the handle of the data acquisition tool, optionallypositioned so that there is a line of sight to a detection camera.

b. Electronic markers such as a local positioning system, andtransducers located at known positions by the imaging system. Thus thetrue position of the acquired data at the image space may be known.

In some embodiments, the use of robotic guidance systems, such as aMazor Robotics product for spine surgery is contemplated. Forco-registration accuracy, the robot may optionally be connected to apatient's bones with screws. The screws, with or without modificationsand/or additions, can serve as markers, for example by detection with acamera and analysis with image processing tools. By knowing a relativelocation of the robot's arm to the screws or other known locations ofthe markers, and by monitoring the markers' location in real time by theimaging system, the location of the robot's arm and tools arepotentially known. Thus it is possible to project an image of therobot's activity inside the body to appear in the actual robot location,within the body.

In some embodiments, a 3D imaging dataset used for producing theholographic image of organs of the patient, may have been obtained whenthe patient was lying down, and the displaying of the internal organsmay be performed when the patient is in a different pose, for example,the patient may be lying on a side or on his stomach.

In some embodiments, a computer used for preparing a display of thepatient's organs optionally deforms a 3D scene for displaying the organsso that the organs appear in a more correct location. The deformation isoptionally performed so that markers' locations in the 3D datasetconform to detectable marker locations in or on the patient's body whenthe patient is in the different pose. By way of a non-limiting example,a computer deforms the 3D dataset so that the markers of the 3D datasetare each exactly at a location of a corresponding detected marker in thepatient. By way of another non-limiting example, the computer deformsthe 3D dataset to minimize a mean-square-error of locations of themarkers in the 3D dataset and locations of the markers detected in or onthe patient in the different pose. By way of yet another non-limitingexample, the computer deforms the 3D dataset according to a mathematicaldeformation model of a deformation suitable for a patient change of posebetween various poses such as: lying on his back, lying on his front,lying on one side or the other, sitting, standing, and so forth.

In some embodiments a holographic image is registered to a real worldbody, with registration marks in a dataset for producing the holographicimage registered to registration marks in or on the real world body, anddisplaying the holographic image is reactive to changes in the realworld body. When the real world body moves, a display of the holographicimage moves accordingly, to remain aligned.

In some embodiments, when the real world body changes shape or scale,the holographic image is recomputed so as to change shape or scaleaccordingly. For example, if the real world object shrinks, the displayof the holographic image is recomputed to shrink as well, such thatregistration marks stay aligned. For example, when the real world objecttwists, the display of the holographic image is computed to twist suchthat the registration marks stay aligned.

In some embodiments, when a real world body has been dissected, thedisplay of the holographic image is recomputed to appear dissected, suchthat the registration marks stay aligned. Dissection is optionallydetected when one or more registration marks are detected to moveapproximately together, in a different direction and/or a differentdistance than one or more different registration marks.

By way of a non-limiting example, a real world body and a correspondingdataset for displaying a holographic image of the real world bodyoptionally have 6 registration marks. In some embodiments when it isdetected that 3 registration marks on the real world object shift in onedirection, maintaining relative distance and/or angles between them, and3 other registration marks on the real world object shift in a seconddirection or rotate some angle, maintaining relative distance and/orangles between them, the holographic image of the real world object isoptionally computed to display as two portions of the real world object,a holographic image displaying each portion still aligned with 3registration marks associated with the image portions.

In some embodiments, less than 3 registration marks, or more than threeregistration marks, an equal number of registration marks or an unequalnumber of registration marks are involved with displaying image portionsof a real world object as if dissected, as described above, or as iftruncated, as will be described below.

In some embodiments, when a real world object is truncated, a display ofthe holographic image is truncated, such that the relevant registrationmarks stay aligned.

In some embodiments an endoscope system is used. Markers are optionallyattached to the end and/or to sections of the endoscope. By knowing therelative location of the endoscope with respect to that of the markers,and by monitoring the markers location in real time by the imagingsystem, the location of the endoscope is potentially known. Thus it ispossible to project an image of the endoscope inside the body to appearat its real coordinates, just as if the physician sees the endoscopeinside the body.

In some embodiments, tools such as endoscopes, catheters or wires whichhave a dynamic shape that changes in real time are involved. In suchembodiments the real time shape with respect to the markers location isoptionally detected, in order to image such a tool, for example anendoscope/catheter/wire, at its real coordinates based on the marker'sposition and tool's shape. Various methods optionally enable detectionof a real time shape of the endoscope/catheter/wire, optionallyincluding a local positioning system (LPS), optical shape detectionthrough optical fibers passing along the tool, or other gadgets alongthe tool. Once the tool is located in an image space of the CGHprojector, the image may optionally be projected in real coordinates.

In some embodiments a camera pill such as the camera pill of “GivenImaging” is involved. The camera pill optionally sends image data andrelative coordinates to its sensor system. By recording the trueposition of the sensor system with respect to the CGH projection system,the true coordinates of the camera pill and its acquired data canoptionally be known. Recording the true position of the sensor systemcan be achieved by placing markers on the sensor system, which istypically a sensor belt. The markers are optionally visible to thedetection camera. As long as the belt is fixed to the patient's anatomythe data may be co registered in the CGH system.

In some embodiments an isotopic imaging (nuclear imaging) system, or PETCT imaging system is involved. An organ is optionally monitored by animage acquiring system that optionally provides data, optionally in itslocal coordinate system. Markers visible to the imaging system areoptionally attached to the skin or connected to the bones/skull. Themarkers are optionally also detectable by the GCH projection unit. Thisenables co-registering the image at the local coordinate system (nuclearimaging system) with the coordinates of the CGH projection unit. Whenimaging data enters the CGH image space it is optionally projected inits true coordinates. Optionally markers are placed on the CT imagingsystem at a location where its relative distance to acquired data forproducing the CGH image is known.

Non Real Time Registration

In some embodiments, such as for procedures involved with non-real timeacquisition of imaging data, for example X-ray, MRI and CT, the capturedCT image coordinates are optionally co-registered to the CGH image spacecoordinates. This is optionally done by stickers and or markers that areattached to a patient's body, on the skin or using screws to bones. Themarkers are such which can be detected by both the CT and the X-rayacquisition system (for example a pattern of metal wires) and can bedetected by the CGH image projection system (for example visible markersor LED patterns).

In some embodiments, such as described above, internal fiducial markersare optionally used. Such markers can be made from titanium or gold orother non-toxic materials. In order for the CGH system to know theirlocation, optionally a near-IR LED in the markers or attached to themarkers can be detected through the skin, or some otherelectronic/acoustic local positioning marker.

In some embodiments, for example for non-real time imaging, the markersare optionally into the bones or skull. For brain surgery, screwing themarkers to the skull and scanning the brain through MRI or CT while themarkers are detectable by the MRI or CT. When a head of a patient isplaced in a CGH image viewing space, sensors detect the markers'real-time location and display a CGH image of the brain such that themarkers are co-registered, and the CGH image shows the brain at itsactual position.

In some embodiments, by inserting a tool at known coordinates, relativeto the CGH imaging system, the physician can view the tool in the brainas if it is open surgery.

In some embodiments, for example involving non-real time imaging, acatheter is optionally placed in a blood vessel related to an organ,close to or inside the organ. A 3D image of the catheter and the organare optionally captured using, for example, CT imaging. The catheter isoptionally fixed, at least temporarily, relative to the organ, andoptionally includes a local positioning system which is co-registeredwith the CGH image projection unit (by placing markers on the unit).When the organ is inside the CGH image space a CGH image of the acquireddata is optionally projected at its actual location.

In some embodiments image processing can optionally replace some or allof the markers, and/or complement the use of markers. By means of imageprocessing of an image of an organ it is possible to co-register MRI,CT, and X-ray images, based on features in the organ. For example, theleg bones acquired in CT can be superimposed by detecting, in the CTdata set, the external aspects of the leg. When the leg is placed withinthe CGH image space, the location or image of the external aspects ofthe leg is optionally detected by the CGH imaging display system and itslocation is calculated. With image processing it is possible to know thelocation of the recorded bone relative to the external aspects of theleg, and the bone is optionally displayed at its true location withinthe leg, including the true depth and orientation.

4D Registration—Time Dependent Registration

In some embodiments a registration method is used that includes spatial,3D position coordinates, and time domain information.

Consider a procedure in which non-real-time 3D data is acquired of asystem that also includes movement, by way of a non-limiting example acyclical motion patterns, such as the lungs, ribs or sternum that areaffected by the breathing cycle. A location of markers that are placedon the ribs and/or sternum changes in a cyclic pattern based on theinhale-exhale cycle.

In some embodiments the markers are detectable by an imaging system suchas MRI or CT at the time of data acquisition, and by the CGH imagedisplay system during projection of the holographic image. The CGH imagedisplay system optionally shifts the holographic image insynchronization with the cyclic phenomena such as: a breathing cycle, toa heartbeat, to cyclical flow, to metabolic processes and electricalsignals such as ECG, which correspond to heartbeats. The co-registrationis maintained for the time evolving movement of the organ.

Optionally, tracking the cyclic movement of a body can be done bymonitoring a time dependent movement of the markers. Knowing therelative position of the acquired data, the CGH image may optionally beco-registered at any time. Image processing can be used to predict atrue location of an organ for different movements of the organ. Forexample, monitoring of the time dependent change in the lungs underregular breathing can be a basis for computing and extrapolating theposition of the lungs, also for deep breathing.

Combining Non Real Time Registration with Real Time Registration

In some embodiments, such as imaging involved with soft tissue such asin of the lungs, some imaging modalities, such as ultrasound forexample, real time data acquisition is not typically used becauseultrasound doesn't image the soft tissue of the lung well. Other 3Dimaging modalities, for example such as, such as CT, which image thesoft tissue better, can optionally be used to detect the soft tissue.Markers are optionally inserted into or attached to the soft tissue. Themarkers are selected to be detectable both by a CT system and anultrasound system. By doing so, the soft tissue organ and the clips maybe co-registered in one local or global coordinate system. By way of anon-limiting example, a patient may be taken to an ultrasound imagingsystem, or similar imaging system which typically involves localpositioning, such as Biosense/Mediguide, that can monitor the positionof the markers in real time and send the position to a CGH imageprojection unit. The coordinates of the ultrasound handle or theBiosense/Mediguide system may optionally be monitored by a guiding unit,potentially providing data for co-registering all the local coordinatesystems (CT and ultrasound) with the CGH image projection unit. When theorgan is in the CGH image viewing space the CGH projection unitoptionally projects the CGH image of the organ and/or the tools withrespect to the clips/markers/beads, at a true location, optionally asbased on global coordinates.

A non-limiting example embodiment involves monitoring blood flow inblood vessels, optionally using a thermal imaging IR camera. The IRcamera captures thermal images of a body, potentially identifying bloodvessels. The IR camera optionally has a known position relative to theCGH image projection unit, or its position is detected by using markersor image processing. In some embodiments, knowing a magnification andfocus, location of the information captured by the IR camera isoptionally co-registered to the CGH image projection unit.

In some embodiments a different image acquisition system is used, suchas a CT system, and co-registers the blood vessels with the desiredorgan, optionally in its local coordinate system.

In some embodiments it is possible to have the IR camera capture imagesin real time, such as the position of the blood vessels, and a globalposition of the organ, based on the CT data, can be calculated. When thebody part or organ is placed in the CGH image viewing space, the CGHprojection unit optionally projects the CGH image at its actuallocation.

In some embodiments a catheter is placed inside a body, and the catheteris used as a marker, using, by way of a non-limiting example, a CTimaging system to monitor the organ+marker/catheter. In some embodimentsa Fluoroscopy (soft X-ray in real time) system is optionally used inreal time monitor the marker/catheter, optionally by using two 2Dprojections to calculate a location. The coordinates of themarker/catheter are optionally registered by the fluoroscopy system,which is optionally co-registered with the CGH image projection system,so that the organ is co-registered with the CGH image.

Combining Real Time Registration of Multiple Acquisition Systems

In some embodiments, 3D data acquired from multiple acquisition systemsis combined to create an image with high resolution information atspecific locations.

In some embodiments the 3D data acquired by a first acquisition systemincludes 3D data of a second tool shape, position and orientation. Sucha combination potentially provides additional information forinterpreting an acquired 3D image.

In some embodiments, a first 3D acquisition system provides a generalview of the heart, and a second 3D acquisition system is inserted into aspecific vein of interest, e.g. the right inferior pulmonary vein. The3D data from both acquisition systems is optionally integrated into one3D image. The combined image displays varying resolutions at variouslocations, or displays other display characteristics in different areasof the image. The combined image optionally includes 3D data of thesecond 3D acquisition system, shape, position and orientation.

Viewing a shape, position and orientation of the data of the secondacquisition system potentially provides additional information andpotentially improves a viewer's understanding of the image of the firstacquisition system. A combined image may potentially be used to guide,or to provide feedback, on how to improve the 3D data acquisition of thesecond system.

In some embodiments, a 3D acquisition system whose field of view isdefined as a cone and of decreasing resolution with distance from thesensor, for example an ultrasound image acquisition system, is used.Viewing a location and orientation of a second system, a viewer canbetter interpret an image and/or reposition the second system to providethe higher resolution and or additional information at a desiredlocation. In some cases, the 3D acquisition system field of view isspherical or semi spherical. In some cases the field of view shape islimited but not defined by simple geometric shapes.

Reference is now made to FIG. 4E, which is a simplified functionalillustration of a display system for displaying a CGH image ofultrasound data according to an example embodiment of the invention.

FIG. 4E shows functional blocks in an example embodiment of theinvention and how they optionally interact in order to display aholographic image of ultrasound data of a body organ, optionally at acorrect location of the body organ within a body.

FIG. 4E depicts a body 211.

An ultrasound imaging system 235 optionally images the body 211,optionally producing a first three-dimensional dataset 231 for producinga computer-generated-holographic (CGH) image of a body organ 210.

The dataset 231 is optionally fed into a computation unit 233, which canproduce a computer generated hologram of the body organ.

A sensor 234 optionally detects locations of features on the ultrasoundimaging system 235 which may optionally serve to align an image of thebody organ with the ultrasound imaging system 235. In some embodimentsthe features are markers 212 a 212 b on the ultrasound imaging system235, which by way of a non-limiting example, may be the ultrasoundhandle. The sensor 234 optionally provides data, for example distanceand direction from the sensor to the markers, to the computation unit233, which can use the data to calculate and produce a computergenerated hologram for displaying the CGH image of the body organ sothat the CGH image of the body organ 210 is aligned and located in acorrect place relative to the ultrasound imaging system 235.

Reference is now made to FIG. 4F, which is a simplified illustration ofa HMD displaying a holographic image of a first object and a visuallyobstructing second object, according to an example embodiment of theinvention.

FIG. 4F shows an HMD 1301 displaying to a viewer 1302 athree-dimensional, optionally holographic image 1303 of the firstobject, for example internal organs, for example liver and related bloodvessels, behind and/or within the visually obstructing second object,for example the skin 1304 of an actual body of the patient 1305 whoseinternal organs are displayed.

FIG. 4F also shows a first marker 1307 attached to the patient's 1305skin 1304. The first marker 1307 is optionally a three-dimensional (3D)marker, and a three dimensional location of the first marker 1307 isoptionally obtained by a location detection system (not shown in FIG.4F) optionally built into the HMD 1301. In some embodiments a markerwith a three-dimensional structure is used, which potentially enablesthe location detection system to locate the first marker 1307 in threedimensions, including depth.

In some embodiments the location detection system optionally detects atleast three points which are not in one line in the 3D first marker1307.

It is noted that detecting three points not in one line potentiallyenables determining a three dimensional position of an object in space,and that when the three points are in one line, the object may berotated about the line as an axis and the three dimensional position ofthe object may change without the three-dimensional location of thethree points being changed.

FIG. 4F also shows a second marker 1308 which is optionally included ina three-dimensional dataset which is used to produce thethree-dimensional, optionally holographic image 1303. The second marker1308 may optionally also be a three-dimensional (3D) marker, and a threedimensional location of the second marker 1308 is optionally obtained bywhatever three dimensional imaging system used to produce thethree-dimensional dataset.

In some embodiments the three dimensional location of the second marker1308 optionally includes at least three points which are not in one linein the 3D second marker 1308.

The HMD 1301, or a computing system (not shown in FIG. 4F) whichprepares three-dimensional images for the HMD 1301, optionally knows arelative position of the first marker 1307 and the second marker 1308,and optionally aligns the three-dimensional image 1303 to the patient's1305 body so as to appear in its correct place within the patient's 1305body under the skin 1304.

In some embodiments the markers are shaped as a three-dimensionalnon-symmetric pyramid. It is noted that marker described with referenceto any other drawing may have a shape of a three-dimensionalnon-symmetric pyramid.

In some embodiments, a single three-dimensional marking optionallyenables detection of both location and orientation of a body/organ/toolattached to or associated with the three-dimensional marking. In someembodiments the three dimensional marking potentially enables detectionof several points, at least some of the points not on a straight line,thereby potentially enabling detection of both location in space andorientation in space.

Reference is now made to FIG. 4G, which is a simplified illustration ofa HMD displaying a holographic image of a first object behind or withina visually obstructing second object, as well as an additional object atleast partially behind or within the visually obstructing second object,according to an example embodiment of the invention.

FIG. 4G shows an HMD 1311 displaying to a viewer 1312 athree-dimensional, optionally holographic image 1313 of the firstobject, for example internal organs, for example liver, behind and/orwithin the visually obstructing second object, for example the skin 1314of an actual body of a patient 1315 whose liver is displayed.

In some embodiments, a system used for obtaining three-dimensional datafor producing the image 1313 is a three-dimensional ultrasound imagingsystem 1316. In some embodiments, a location detection system (notshown) optionally built into the HMD 1311 locates a three-dimensionallocation of the ultrasound imaging system 1316. In some embodiments, thelocation detection system optionally locates a three-dimensionallocation and orientation of the ultrasound imaging system 1316 based ona marker 1317 optionally attached and/or built into the ultrasoundimaging system 1316. In some embodiments, the marker 1317 is athree-dimensional marker as described above.

In some embodiments, the ultrasound imaging system 1316 provides adataset for producing the image 1313, optionally in real-time. Since theHMD 1311 optionally detects the three-dimensional location andorientation of the ultrasound imaging system 1316, and receives datafrom the ultrasound imaging system 1316 for displaying the threedimensional image 1313, the HMD 1311 can display the image 1313 in itscorrect location in space, based on a relative location and orientationof the ultrasound imaging system 1316 relative to the HMD 1311, and therelative location and orientation of the first object relative to theultrasound imaging system 1316.

FIG. 4G also shows a tool 1319, such as a syringe, which is partlyoutside the patient's 1315 skin 1314, and is thus optionally seen by theviewer 1312 through the HMD 1311. In some embodiments the syringeincludes a needle, a first part 1320 a of which is outside the skin1314, and is viewed directly by the viewer 1312, and a second part 1320b that is hidden from unaided view by the patient's 1315 skin 1314, andis viewed as part of the image displayed by the HMD 1311, as captured bythe ultrasound imaging system 1316. The first part 1320 a of the needleis optionally viewed as a continuation of the second part 1320 b of theneedle, since the image displayed by the HMD 1311 is correctly aligned,or registered, in space, as described above.

Reference is now made to FIG. 4H, which is a simplified illustration ofan imaging system 1316 for obtaining three-dimensional data fordisplaying a 3D image of internal organs in a body, according to anexample embodiment of the invention.

FIG. 4H shows an image acquisition system 1316, by way of a non-limitingexample an ultrasound system, and a marker 1317 attached to the imageacquisition system 1316, which is optionally capturing an image orimages below a patient's skin 1314. In some embodiments the marker 1317is a three-dimensional marker as described above.

The marker 1317 potentially enables a location system to detect alocation and orientation of the marker 1317 in space. From the locationand orientation of the marker 1317 in space a location and orientationof the image acquisition system 1316 in space may be calculated. Fromthe location of the image acquisition system 1316 in space, a locationand orientation of objects in a 3D image captured by the imageacquisition system 1316 may be calculated. Regardless of whether thelocation of the marker 1317 is detected by an HMD, as shown in FIG. 4G,or by some other location detection system, the location in space of theobjects captured by the 3D imaging system 1316 can be calculated.

In some embodiments the 3D image acquisition system 1316 is a real timeimage acquisition system, and 3D images displayed based on data capturedby the 3D image acquisition system 1316 are optionally displayed inreal-time, that is, within a fraction of a second of capturing the 3Dimages.

Some non-limiting example embodiments of 3D image acquisition system towhich 3D markers can be attached include ultrasound transducers, TEEimaging transducers, CT imaging systems, and additional systems aslisted herein.

Reference is now made to FIG. 4I, which is a simplified illustration ofan image acquisition system 1316 for obtaining three-dimensional datafor displaying a 3D image of internal organs in a body, according to anexample embodiment of the invention.

FIG. 4I shows an image acquisition system 1316, by way of a non-limitingexample an ultrasound system, and a first marker 1318 viewable by anexternal location detection system (not shown). The external locationdetection system may be on a HMD such as shown in FIG. 4G, or elsewherewhich can view or detect and measure a location of the first marker1318.

FIG. 4I also shows a second marker 1319, somewhere within a patient'sbody, hidden by skin 1314, yet detectable and its location measurable atleast by the imaging system 1316.

In some embodiments, locations of both the first marker 1318 and thesecond marker 1319 are detectable and/or measurable by theabove-mentioned location detection system (not shown), or by some otherlocation detection system. The location detection system which detectsboth the first marker 1318 and the second marker 1319 optionallyprovides a location and orientation of the first marker 1318 relative tothe second marker 1319. A display system which is provided with thelocation of the first marker 1318, a relative location of the secondmarker 1319, and a dataset of internal organs (not shown) which includesdata describing the second marker 1319, optionally displays the internalorgans in a correct position in space, based on calculating the internalorgan location relative to the second marker 1319, the location of thesecond marker 1319 relative to the first marker 1318, and the locationof the first marker relative to the patient's body and/or relative tothe display system and/or relative to the viewer.

In some embodiments, locations of both the first marker 1318 and thesecond marker 1319 are detectable and/or measurable by the imageacquisition system 1316. The image acquisition system 1316 optionallyprovides an image of internal organs (not shown) which includes imagesand/or locations of the first marker 1318 and the second marker 1319. Athree-dimensional display system which can view and measure a locationof at least the first marker 1318 can display an image with the firstmarker located correctly in space, and use a dataset of the internalorgans and the two markers from the imaging system 1316 to produce anddisplay a 3D image of the internal organs located correctly in space.

Reference is now made to FIG. 4J, which is a simplified illustration ofan image acquisition system 1316 and an additional object, according toan example embodiment of the invention.

FIG. 4J shows an image acquisition system 1316, by way of a non-limitingexample an ultrasound system, and an additional object, by way of anon-limiting example a surgical tool 1322.

A first marker 1317, viewable and its location detectable and/ormeasurable by an external location detection system (not shown), isoptionally attached to, marked upon, or built into the imaging system1316. A second marker 1321, viewable and its location detectable and/ormeasurable by the external location detection system is optionallyattached to, marked upon, or built into the surgical tool 1322.

By way of a non-limiting example, a first portion 1322 a of the surgicaltool 1322 is external to a skin 1314 of a patient's body, and a secondportion 1322 b of the surgical tool 1322 is within the patient's body,hidden from unaided view by the skin 1314.

The image acquisition system 1316 optionally provides data for producing3D images to a three-dimensional display system (not shown), fordisplaying internal organs, hidden beneath the skin 1314. By virtue of alocation of the first marker 1317 being detectable and measurable by theexternal location detection system, and by virtue of geometricproperties of the imaging system 1316 being known, an image of theinternal organs is optionally shown registered to the patient's body, ata correct location in space.

In some embodiments the imaging system 1316 optionally provides data forproducing 3D images to a three-dimensional display system (not shown),including data for displaying the second portion 1322 b of the surgicaltool 1322. By virtue of the location of the second marker 1321 beingdetectable and measurable by the external location detection system, andby virtue of geometric properties of the surgical tool 1322 being known,an image of the second portion 1322 b is optionally shown registered tothe patient's body, at a correct location in space.

In some embodiments the imaging system 1316 optionally provides data forproducing 3D images to a three-dimensional display system (not shown),including data for displaying the second portion 1322 b of the surgicaltool 1322. By virtue of the location of the second marker 1321 beingdetectable and measurable by the external location detection system, andby virtue of geometric properties of the surgical tool 1322 being known,an image of the entire surgical tool 1322 is shown, including the secondportion 1322 b, which is inside the body, and the first portion 1322 a,which is outside the body. Both portions 1322 a 1322 b of the surgicaltool are optionally shown registered to the patient's body, at a correctlocation in space.

In some embodiments the three-dimensional display system (not shown),displays images of entire objects which are partly hidden by otherobjects and partly visible. Both the hidden part(s) and the visiblepart(s) are shown in their correct position in space.

In some embodiments the three-dimensional display system (not shown),displays digital images which are partly hidden by other objects andpartly visible. Both the hidden part(s) and the visible part(s) areshown in their correct position in space.

In some embodiments the image acquisition system 1316 optionallyprovides data for producing 3D images to a three-dimensional displaysystem (not shown), including data for displaying the second portion1322 b of the surgical tool 1322. By virtue of the location of the firstmarker 1317 being detectable and measurable by the external locationdetection system, and by virtue of geometric properties of the imagingsystem 1316 being known, an image of the second portion 1322 b which maybe provided to the display system, optionally in real-time, isoptionally shown registered to the patient's body, at a correct locationin space.

In some embodiments there is optionally a marker (not shown), optionallyattached to, marked upon, or built into the second portion 1322 b of thesurgical tool 1322, optionally detectable by the image acquisitionsystem 1316.

Example of Not-Necessarily Medical Embodiments

An aspect of some embodiments of the invention involves displaying, byway of a non-limiting example, a CGH image of hidden components,correctly placed within a view of a real world scene. In someembodiments, such displaying is optionally achieved by the displaydevice locating itself relative to the real world scene, and displayingthe CGH image of the hidden components at their correct location.

Reference is now made to FIG. 5A, which is a simplified illustration ofa HMD displaying a holographic image of a first object behind or withina visually obstructing second object, according to an example embodimentof the invention.

FIG. 5A depicts the holographic image of the first object, for examplenetwork wiring 215, or electric wiring, or water pipes 225, or airconditioning ducts, or house frame joists, behind or within the visuallyobstructing second object, for example the wall 216 of a room.

FIG. 5A depicts a viewer 201 wearing a HMD 202, which displays thewiring 215 and the pipes 225 in the room.

The viewer 201 also sees the walls 216 of the room, so sees both thewiring 215 and pipes 225 and the walls 216 of the room.

In some embodiments, the wiring 215 and the pipes 225 are correctlylocated in space relative to the walls 216.

In some embodiments the display system, the HMD 202 for example, obtainsa first relative location or coordinates and/or orientation of one ormore marking(s) 217 on the wall 216, relative to the HMD 202. The HMD202 also obtains a three-dimensional dataset describing the wiring 215and/or the pipes 225, and optionally displays the wiring 215 and/or thepipes 225 in their correct location behind or within the walls 216.

In some embodiments the display system, the HMD 202 for example, obtainsa first relative location or coordinates of a known portion of thewiring 215 and/or the pipes 225, such as a location of one or more wallplate(s) 218 a and/or 218 b, relative to the HMD 202. The HMD 202 alsooptionally obtains a three-dimensional dataset describing the wiring 215and/or the pipes 225, and optionally displays the wiring 215 and/or thepipes 225 in their correct location behind or within the walls 216.

In some embodiments, the HMD 202 displays the holographic image of thewiring 215 and/or the pipes 225 so that the marking(s) 217 and/or theknown portion 218 a 218 b locations in the holographic image coincide,also termed herein co-register, with the marking(s) 217 and/or the knownportions 218 a 218 b locations in the real world, on the walls 216 ofthe room. Such co-registration potentially displays the wiring 215and/or the pipes 225 in the correct location relative to the walls 216.

In some embodiments the HMD 202 optionally has one or more sensor(s) 204which can detect and locate the marking(s) 217 and/or the known portions218 a 218 b. The sensor 204 optionally measures distance and/or angletoward the marking(s) 217 and/or the known portions 218 a 218 b on thewalls 216. The measurement potentially enables the HMD 202 to determinea location of the walls 216 relative to the displayed wiring 21 and/orpipes 2255.

The sensor 204 may optionally be any one of the sensors describedherein.

The marking(s) 217 may optionally be any of the markings describedherein.

In some embodiments, tracking the wiring 215 and/or pipes 225 locationand/or orientation in space relative to the HMD 202 is optionallyperformed by an external system tracking the HMD 202.

In some embodiments, tracking a display's orientation in space, such asthe HMD 202, is optionally performed by the display itself, by opticallytracking location of objects, external to the display, in space; byoptically tracking specific markings in a vicinity of the display inspace; by using direction finding similarly to direction finding bysmart phones; by using an accelerometer; and by using a gravity sensor.

In some example embodiments, a view of wiring and/or pipes within awall, by way of a non-limiting example as described with reference toFIG. 5A, may be done, by way of a non-limiting example, for planningwork on the walls 216 or cutting into the walls 216.

In some non-limiting example embodiments, only a specific section ofwiring and/or pipes, such as section 227 depicted in FIG. 5A, isdepicted behind the wall 216.

In some embodiments a 3D structure of elements of the scene which arepartially hidden from naked-eye view is optionally known, for example asa CAD dataset of the elements, or as some other dataset describing astructure of the hidden elements. In such embodiments the HMD 202optionally measures a location of reference elements which are visibleto the HMD 202, or at least susceptible to location measurement by theHMD 202, and displays the scene, including the elements which are hiddenfrom naked-eye view, correctly aligned with the reference elements andso correctly aligned in the real world.

The terms “dataset” and “imaging data” in all their grammatical formsare used throughout the present specification and claims to mean adataset for producing a three-dimension image, including medical imagingdatasets, Computer Aided Design (CAD) datasets, mapping datasets,geographic datasets, geologic datasets, and additional datasets whichcontain three-dimensional data.

In some embodiments, a display such as described herein displays CADdata for engineering review of a working mechanism, such as a motor. Insome embodiments a viewer can optionally see inner parts of a mechanismdynamically, in motion, optionally located correctly relative to anouter envelope of the mechanism. In some embodiments the display enablesviewing the inner parts of the mechanism from various angles, optionallyincluding a full 360 degree walk-around and/or top-and-bottomcircumnavigation, with capability to observe the inner parts of themechanism.

By way of some non-limiting examples a viewer may view inner workings ofa motor and/or motor parts within a car.

In some embodiments such displaying is optionally used for structuredesign, for teaching, in medical scenarios and in non-medical scenarios.

Reference is now made to FIGS. 5B and 5C, which are simplifiedillustrations of a specific portion 227 of FIG. 5A, in which a HMD isdisplaying a holographic image of pipes 225 f 225 b 225 x behind avisually obstructing wall, according to an example embodiment of theinvention.

FIG. 5B depicts the portion 227 with various pipes behind the wall. Somepipes 225 f pass in front of some other pipes 225 b 225 x. By way of anon-limiting example it is desired to drill a specific pipe 225 x whichpasses behind some of the pipes 225 f and in front of some of the pipes225 b.

FIG. 5B also depicts, by way of a non-limiting example, a drill 230directed toward the specific pipe 225 x. FIG. 5B depicts that a viewermay optionally have a view of both a CGH image of pipes and a realobject, the drill 230.

In some embodiments, as depicted, by way of a non-limiting example inFIG. 5B, a CGH image of a guide line 232 may optionally be displayedcontinuing the direction of the drill 230, showing where the drill wouldintersect the specific pipe 225 x.

FIG. 5C depicts the portion 227 with the various pipes 225 b 225 f 225 xbehind the wall.

FIG. 5B also depicts the drill 230 drilling the specific pipe 225 x. thedrill 230 is optionally directed at an angle so as to optionally drill alocation behind the front pipes 225 f. FIG. 5B depicts that a viewer mayoptionally naturally direct a drill base on seeing the pipes 225 x 225 b225 f as they are in their correct location in space.

Viewing Arc-Orbit

In some embodiments it is desirable that the physician be able to seesurgical tools and/or an organ that are inside the patient. An aspect ofsuch a capability is termed the viewing arc. The viewing arc aspectimplies that the physician is able see a 3D image from different viewingpositions, as if he is traversing on an arc, or moving around the organfrom a choice of any angle. Consider a surgeon inspecting an organ priorto or during surgery. It is potentially beneficial that the physician beable to see a proximity and relative position of tissue to vessels,behind, within and to the sides of the organ, relative to thephysician's vantage point.

A CGH image provides the physician with a natural and intuitive viewingarc which can potentially provide spatial comprehension. The physiciancan optionally walk around the CGH image, and view the CGH image fromdifferent angles, naturally and intuitively. The physician does notrotate the CGH image on a display. The CGH image of the organ isco-registered with the patient's body, and as the physician views theorgan and moves around the organ he views the co-registered image andorgan from different angles.

The CGH image potentially displays spatial resolution of less than 1 mm,2 mm, 5 mm or 10 mm. The term arc in this context is used for a changeof viewing angle or position relative to the CGH image it is notnecessarily a geometric arc.

In some embodiments a needle like tool is navigated: to determine a bestend expiration depth. See the figures below—the light area displaysmetastasis in a liver surrounded by blood vessels. In the left figure,which shows a lateral view, no clear path for a needle to reach themetastasis without damaging the vessels is seen. By changing the viewingposition to a more anterior-posterior projection, shown in the rightfigure, the relative 3D position of the blood vessels to the metastasisis clarified, demonstrating a route to the metastasis without the riskof damaging a blood vessel. The motion path between the lateral and APviewing position is termed herein the viewing arc-orbit.

A lateral view might show the metastasis and the blood vessels crowdedin space while an anterior-posterior view may present a differentspatial configuration. In the figures below, the anterior-posterior viewshows larger distances between the metastasis and the blood vessels.

Reference is now made to FIG. 6A, which is a simplified line drawing 601of blood vessels 602 and metastasis locations 603 a 603 b according toan example embodiment of the invention.

FIG. 6A depicts the blood vessels 602 and metastasis locations 603 a 603b as viewed from a lateral direction.

In some example embodiments a surgeon may move his display, changing hispoint of view, relative to an actual location of the body of thepatient, specifically relative to an actual location of the bloodvessels 602 and metastasis locations 603 a 603 b, so as to look at apatient's body from various directions.

Reference is now made to FIG. 6B, which is a simplified line drawing 605of the blood vessels 602 and the metastasis locations 603 a 603 b ofFIG. 6A, according to an example embodiment of the invention.

FIG. 6B depicts the blood vessels 602 and metastasis locations 603 a 603b as viewed from an anterior-posterior direction.

The example of FIG. 6B illustrates that when the surgeon changes his/herdirection of viewing the patient's body, the CGH image displayoptionally provides the surgeon with a CGH image correctly located andviewed from the surgeon's new viewing direction, which may sometimesprovide a clearer, more separated in the image, view of the bloodvessels 602 and metastasis locations 603 a 603 b.

In some embodiments support for a change in viewing direction such asshown in the difference between FIG. 6A and FIG. 6B is provided, withthe displayed blood vessels 602 and metastasis locations 603 a 603 bremaining correctly registered within a patient's body. Such supportpotentially enables physicians a natural interface for inspecting apatient and planning medical intervention.

Clinical Applications

In some embodiments, 3D see-through vision is used in image guidedtherapy, for example for irreversible electroporation (IRE) ablation.IRE procedures are applied in tumor ablation in regions where precisionand conservation of blood vessels and nerves are of importance.

In some embodiments, multiple electrodes, shaped in the form of longneedles, are placed around a target tumor. The point of penetration forthe electrodes is chosen according to anatomical considerations. Imagingis potentially essential to the placement of the needles that should beplaced with high precision. The electrode needles are preferablyparallel to each other and placed around the tumor at a high precisionof 1 mm, 2 mm, 5 mm. This precision also relates to the depth that eachneedle is inserted relative to each other and relative to the tumor.

Reference is now made to FIG. 7A, which is a simplified isometric linedrawing illustration of needles and a tumor with a specific body volumeaccording to an example embodiment of the invention.

An example embodiment illustrated by FIG. 7A may be a procedure ofIrreversible Electropolation (IRE), which potentially benefits fromparallelism and similar/identical depth of needles, with an accuracy of1 mm.

FIG. 7A depicts the needles 701, the tumor 702 and the specific bodyvolume 703 as viewed from a surgeon's viewpoint.

FIG. 7A depicts an example embodiment of a CGH image of the tumor 702and the specific body volume 703, co-registered and displaying a CGHimage of the needles 701 at a planned location.

Reference is now made to FIG. 7B, which is a simplified isometric linedrawing illustration of needles and a tumor with a specific body volumeaccording to yet another example embodiment of the invention.

FIG. 7B depicts the needles 705, the tumor 702 and the specific bodyvolume 703 as viewed from a surgeon's viewpoint, in an actual CGH image,displaying the actual needles 705 at their actual location, as acquiredby a 3D image acquisition system.

FIG. 7B depicts an example embodiment of a CGH image of the tumor 702and the specific body volume 703, in some embodiments not necessarilyco-registered with any other image, since FIG. 7B depicts a CGH imageoptionally produced based on a single 3D data set, optionally acquiredby one data acquisition system. However, in some embodiments the CGHimage is co-registered with the patient's body (not shown), and shown atits actual location in real space.

It is noted the in embodiments involving an IRE procedure, it is desiredthat the needles be parallel, that tips of the needles form a plane, andthat the tips be approximately equally spaced from a metastasis ortumor.

The example embodiment depicted in FIGS. 7A and 7B potentially enable aphysician to visually confirm that the needles 705 and the needle tipsare at a desired location, rather than estimating the location of theneedles and needle tips based, for example, on measuring their depthinto the patient body. The physician potentially saves time in theprocedure, potentially shortening the procedure, which can potentiallybenefit the patient and the physician. The patient may be saved fromexcess radiation by viewing needle tips at their location within thebody, rather than performing x-ray imaging to verify the needle tiplocation(s).

Kidney Transplant, Laparoscopic Kidney Removal or LaparoscopicNephrectomy

In some embodiments, such as embodiments involving pre-surgicalplanning, markers are optionally placed on a donated kidney. Internalmarkers are optionally placed on an artery that supplies blood tokidneys, and optionally on the vein that carries blood away from thekidney. Optionally, external markers are placed on the recipient'schest, by way of some non-limiting examples by screws to the ribs orstickers on the abdomen.

The CGH image display system and the 3D acquisition system optionallydetect the markers on the recipient. Image registration of the image isoptionally obtained using some or all of the above-mentioned themarkers.

The 3D image acquisition system optionally detects the internal markerswithin the recipient as well as the markers on a transplanted kidney,once the kidney is in the recipient's body. The internal markers andmarkers on the kidney optionally assist a physician using 3D see-throughvision in positioning the kidney.

An Example Clinical Application Method

In some embodiments, 3D see-through vision is used to support methodsand tools to enable direct rapid puncture during lung biopsy surgicalprocedures, potentially reducing needle related pleural damage,improving biopsy quality and requiring less imaging time, hencepotentially less radiation.

In some embodiments, a method of using 3D see-through vision in clinicalapplications includes the following steps:

-   -   Apply markers: to the patient and/or organs and or additional        tools    -   Image the patient and markers with a 3D data acquisition system    -   Co-Register the CGH image to the patient location    -   Plan the surgical activities    -   Proceed with the plan

The above method potentially enables direct rapid puncture, potentiallyreduces needle related pleural damage, potentially improves biopsyquality and potentially requires less imaging time, hence potentiallyreduces radiation.

In some embodiments image guided therapies such as, by way of somenon-limiting examples, lung biopsy, ablation therapies and minimallyinvasive spinal surgery use 3D see-through vision. The above-mentionedexample image guided therapies optionally include navigations onco-registered images with a LPS and GPS tracking system, where apotentially significant aspect is true depth registration allowing for awide viewing orbit to visualize and plan point and direction of entry,and/or optionally tracking the intervention until a correct depth isachieved, while being able to view the spatial relationships.

In some embodiments, in order to visualize and visually track the needlein the holographic image as it is navigated to the target tissue,real-time automatic registration of the needle to custom fiducialmarkers positioned onto the surface of a patient, abdomen or thorax orother area, as relevant for the procedure, is optionally used.

In some embodiments real time registration is used to accommodate forphysiological motion of organs, e.g. the abdomen. A patient isoptionally imaged with a 3D imaging modality while the patient hasfiducial markers positioned onto the body. The fiducial markersoptionally remain fixed to the body during the procedure and willoptionally be used to register and fuse a 3D see-through image onto thepatient's anatomy.

In some embodiments a biopsy needle navigation is tracked, during aprocedure, in a 3D see-through image that is generated, optionally by anelectromagnetic (or similar) tracking system. In an electromagnetictracking system a local electromagnetic field is optionally generatedaround an operating table and the needle tip optionally includes a coilthat can be tracked within the electromagnetic field. The holographicimage system is optionally able to track the movement of the needle andpresent the location of the needle at any given time and at any givenlocation within a holographic image. The holographic image may be a 3Dsee-through image or a 3D holographic image that is displayed floatingabove the patient.

Example Methods

An example embodiment method for displaying an interference basedholographic image of an inner body organ within a body, providing botheye convergence and eye focus accommodation cues is now described.

Reference is now made to FIG. 8A, which is a simplified flow chartillustration of a method for displaying an interference basedholographic image of an inner body organ within a body, providing botheye convergence and eye focus accommodation cues, according to anexample embodiment of the invention.

The method depicted by FIG. 8A includes:

obtaining a first three-dimensional dataset comprising data forproducing a computer-generated-holographic (CGH) image of the inner bodyorgan (802);

detecting a location of a first registration location in the inner bodyorgan (804). In some embodiments the detecting the location of the firstregistration location includes detecting the location of the firstregistration location in the 3D dataset;

detecting a location of a second registration location on the body(806). In some embodiments the detecting the location of the secondregistration location on the body includes detecting by a CGH projectionunit. In some embodiments a relative location of the first registrationlocation with respect to the second registration location is known. Insome embodiments the CGH image is calculated so that the location of thefirst registration location with respect to the second registrationlocation is co-registered;

producing the CGH image of the inner body organ (808); and

displaying the CGH image of the inner body organ (810);

wherein the displaying the CGH image of the inner body organ comprisesdisplaying the CGH image of the inner body organ so that the firstregistration location is displayed at a specific spatial locationrelative to the second registration location. In some embodiments thefirst registration location is optionally detected in or on the firstthree-dimensional dataset.

In some embodiments the order of detecting the location of the secondregistration location (806) and the producing the CGH image of the innerbody organ (808) may optionally be reversed.

In some embodiments the second registration location on the body isoptionally detected in or on the body by a system for displaying the CGHimage of the inner body organ, and/or by a detection systemco-registered with the system for displaying the CGH image of the innerbody organ.

In some embodiments a relative position and/or orientation of the firstregistration location and the second registration location is known.

In some embodiments the location of the first registration location withrespect to the second registration location in known by at least one thefollowing methods:

1. Both registration locations are part of a single solid body withknown geometry.

2. Both registration locations are located by a monitoring system usingsensors such as acoustic and/or mechanical and/or electromagneticsensors.

In some embodiments aligning the CGH image of the inner body organ tothe body is performed by a viewer translating and/or rotating and/orscaling the CGH image to align to the body. The translating and/orrotating and/or scaling are optionally performed usingman-machine-interface commands, the viewer optionally judging alignmentby eye, viewing both the CGH image and the real body, and optionallyacting on depth cues such as eye convergence and eye focus.

Reference is now made to FIG. 8B, which is a simplified flowchartillustration of a method for displaying a holographic image of a bodyorgan at a correct location of the body organ within a body, accordingto an example embodiment of the invention.

The method of FIG. 8B, includes:

obtaining a first three-dimensional dataset including data for producinga computer-generated-holographic (CGH) image of the body organ (822);

determining a location of at least one first registration location inthe body organ (824);

detecting a location of at least one second registration location on thebody (826);

producing an interference based CGH image of the body organ (828); and

displaying the CGH image of the body organ (830).

In some embodiments the above method optionally includes knowing arelative distance and/or orientation between the first and secondregistration markers.

In some embodiments the displaying the CGH image of the body organ is adisplaying of the CGH image of the body organ so that the firstregistration location is displayed at a specific spatial locationrelative to the second registration location, so that the CGH image ofthe body organ is aligned and located in a correct place of the bodyorgan relative to the body.

In some embodiments the CGH image of the body organ provides a viewerwith both eye convergence and eye focus depth cues.

An example embodiment method for displaying an interference basedholographic image of a first object behind or within a visuallyobstructing second object, providing both eye convergence and eye focusaccommodation cues is now described.

Reference is now made to FIG. 9 , which is a simplified flow chartillustration of a method for displaying an interference basedholographic image of a first object behind or within a visuallyobstructing second object, providing both eye convergence and eye focusaccommodation cues, according to an example embodiment of the invention.

The method depicted by FIG. 9 includes:

obtaining a first three-dimensional dataset comprising data forproducing a computer-generated-holographic (CGH) image of the firstobject (902);

detecting a location of a first registration location in the firstobject (904). In some embodiments the first registration location ispart of the three-dimensional dataset for producing the CGH image of thefirst object. In some embodiments the location of a first markeroptionally includes orientation;

detecting a location of a second registration location in the secondobject (906). In some embodiments the location of a second markeroptionally includes orientation. In some embodiments the detecting thelocation of a second registration is optionally performed using adetection sensor as part of a CGH projection unit;

producing the CGH image of the first object (908); and

displaying the CGH image of the first object (910);

wherein the displaying the CGH image of the first object comprisesdisplaying the CGH image of the first object so that the firstregistration location in the first object is located at a specificspatial location relative to the second registration location.

In some embodiments the first registration location is optionallydetected in a three-dimensional dataset for producing a CGH image of thefirst object.

In some embodiments the second registration location on the body isoptionally detected in or on the second object by a system fordisplaying the CGH image of the first object, and/or by a detectionsystem co-registered with the system for displaying the CGH image of thefirst object.

In some embodiments a relative position of the first registrationlocation and the second registration location is known.

In some embodiments a relative location and/or orientation of a firstmarker with respect to a second marker is detected by a detection unitusing one of the mentioned sensory technologies.

Reference is now made to FIG. 10 , which is a simplified block diagramillustration of apparatus for displaying an interference basedholographic image of a first object behind or within a visuallyobstructing second object, providing both eye convergence and eye focusaccommodation cues, according to an example embodiment of the invention.

FIG. 10 depicts a computation unit 1002 receiving a firstthree-dimensional dataset 1004 including data for producing acomputer-generated-holographic (CGH) image 1006 of the first object. Insome embodiments the computation unit 1002 optionally receives alocation of a first registration location in the first object.

In some embodiments the computation unit 1002 optionally detects alocation of a first registration location in the first object, andproduces the CGH image 1006 of the first object. As mentioned above, insome embodiments the computation unit 1002 optionally receives thelocation of a first registration location in the first object.

FIG. 10 also depicts a sensor 1008 for detecting a location of a secondregistration location in the second object, and a CGH image display 1010for displaying the CGH image of the first object.

In some embodiments, the relative location of second registrationlocation in the second object with respect to the first registrationlocation in the first object is known to the computation system. In someembodiments, the two registration locations markers are detected by anadditional detection system (not shown) and the relative location issent to the computation system.

In some embodiments the displaying the CGH image of the first objectoptionally includes displaying the CGH image of the first object so thatthe first registration location in the first object is located at aspecific spatial location relative to the second registration location.

In some embodiments, the specific spatial location optionally includesdisplaying the CGH image of the first object at its actual spatiallocation relative to the second object, in a way which appears that thefirst object appears to a viewer to be inside, within, or behind thesecond object.

Reference is now made to FIG. 11 , which is a simplified flow chartillustration of a method for displaying an image of an object acquiredusing a first coordinate system by a CGH projection unit using a secondcoordinate system co-registered to the first coordinate system accordingto an example embodiment of the invention.

The method of FIG. 11 includes:

a. providing a CGH image projection unit that monitors its display space(1102);

b. attaching to the object markers that are detectable in both the firstand the second coordinate systems (1104);

c. capturing an image of the object with the markers using the firstcoordinate system (1106);

d. detecting the markers by the CGH projection unit using the secondcoordinate system (1108);

e. calculating a position of the object in the second coordinate system(1110), and

f. projecting the CGH image of the object at a location based on theposition of the object in the second coordinate system (1112).

Reference is now made to FIG. 12 , which is a simplified flow chartillustration of a method for co-registration of an image of an objectacquired at a first coordinate system to a CGH projection unit at asecond coordinate system according to an example embodiment of theinvention.

The method of FIG. 12 includes:

a. providing a CGH projection unit that monitors its display space(1202);

b. attaching to the object markers that are detectable by the CGHprojection unit in the second coordinate system (1204);

c. capturing the image of the object and the markers using the firstcoordinate system (1206);

d. sending the image of the object using the first coordinate system tothe CGH projection unit (1208);

e. using the CGH projection unit to detect the markers (1210);

f. calculating the position of the image of the object using the secondcoordinate system (1212); and

g. projecting the image of the object based on the calculating theposition of the image of the object using the second coordinate system(1214).

In some embodiments a shape and location of an additional object such asa tool, a robot's arm, a catheter, an endoscope are also sent to the CGHprojection unit, using either the first coordinate system or the secondcoordinate system, or even some other coordinate system.

It is expected that during the life of a patent maturing from thisapplication many relevant image acquisition systems will be developedand the scope of the term image acquisition system is intended toinclude all such new technologies a priori.

It is expected that during the life of a patent maturing from thisapplication many relevant holographic image display systems will bedeveloped and the scope of the term holographic image display system isintended to include all such new technologies a priori.

As used herein the term “about” refers to ±10%.

The terms “comprising”, “including”, “having” and their conjugates mean“including but not limited to”.

The term “consisting of” is intended to mean “including and limited to”.

The term “consisting essentially of” means that the composition, methodor structure may include additional ingredients, steps and/or parts, butonly if the additional ingredients, steps and/or parts do not materiallyalter the basic and novel characteristics of the claimed composition,method or structure.

As used herein, the singular form “a”, “an” and “the” include pluralreferences unless the context clearly dictates otherwise. For example,the term “a unit” or “at least one unit” may include a plurality ofunits, including combinations thereof.

The words “example” and “exemplary” are used herein to mean “serving asan example, instance or illustration”. Any embodiment described as an“example or “exemplary” is not necessarily to be construed as preferredor advantageous over other embodiments and/or to exclude theincorporation of features from other embodiments.

The word “optionally” is used herein to mean “is provided in someembodiments and not provided in other embodiments”. Any particularembodiment of the invention may include a plurality of “optional”features unless such features conflict.

Throughout this application, various embodiments of this invention maybe presented in a range format. It should be understood that thedescription in range format is merely for convenience and brevity andshould not be construed as an inflexible limitation on the scope of theinvention. Accordingly, the description of a range should be consideredto have specifically disclosed all the possible sub-ranges as well asindividual numerical values within that range. For example, descriptionof a range such as from 1 to 6 should be considered to have specificallydisclosed sub-ranges such as from 1 to 3, from 1 to 4, from 1 to 5, from2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numberswithin that range, for example, 1, 2, 3, 4, 5, and 6. This appliesregardless of the breadth of the range.

Whenever a numerical range is indicated herein, it is meant to includeany cited numeral (fractional or integral) within the indicated range.The phrases “ranging/ranges between” a first indicate number and asecond indicate number and “ranging/ranges from” a first indicate number“to” a second indicate number are used herein interchangeably and aremeant to include the first and second indicated numbers and all thefractional and integral numerals therebetween.

It is appreciated that certain features of the invention, which are, forclarity, described in the context of separate embodiments, may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the invention, which are, for brevity, described in thecontext of a single embodiment, may also be provided separately or inany suitable sub-combination or as suitable in any other describedembodiment of the invention. Certain features described in the contextof various embodiments are not to be considered essential features ofthose embodiments, unless the embodiment is inoperative without thoseelements.

Although the invention has been described in conjunction with specificembodiments thereof, it is evident that many alternatives, modificationsand variations will be apparent to those skilled in the art.Accordingly, it is intended to embrace all such alternatives,modifications and variations that fall within the spirit and broad scopeof the appended claims.

All publications, patents and patent applications mentioned in thisspecification are herein incorporated in their entirety by referenceinto the specification, to the same extent as if each individualpublication, patent or patent application was specifically andindividually indicated to be incorporated herein by reference. Inaddition, citation or identification of any reference in thisapplication shall not be construed as an admission that such referenceis available as prior art to the present invention. To the extent thatsection headings are used, they should not be construed as necessarilylimiting. In addition, any priority document(s) of this applicationis/are hereby incorporated herein by reference in its/their entirety.

What is claimed is:
 1. A system for displaying a three-dimensional (3D)image of a virtual object in a correct location relative to a realobject, comprising: a display for displaying a 3D image of the virtualobject; a circuit for measuring and determining a location of a featureon the real object relative to the display; the circuit adapted for useby the display for displaying the 3D image of the virtual object;wherein the circuit is adapted to compute the data for displaying the 3Dimage of the virtual object in the correct location relative to the realobject; and the display is configured to display a plurality ofdifferent points in the 3D image of the virtual object as focused at aplurality of respective different distances from a viewer using thedisplay using the measuring and the determining.
 2. A system accordingto claim 1, wherein the display comprises a Head Mounted Display (HMD).3. A system according to claim 1 in which the display is adapted todisplay the 3D image of the virtual object focused at a same distancefrom the viewer as the real object, based on the location measurementunit determining the location of a feature of the real object, therebyenabling the viewer to view both the 3D image of the virtual object andthe real object in focus simultaneously.
 4. A system according to claim1 in which the computing unit is adapted to compute the data fordisplaying the 3D image of the object in the correct location relativeto the real object based on measuring the location on the surface of thereal object.
 5. A system according to claim 1 in which the 3D imagepotentially appears floating in space, enabling the viewer to selectwhat to view in focus, the 3D image or the real object.
 6. A systemaccording to claim 1 in which the location measurement system iscomprised in the display.
 7. A system according to claim 1 in which thecomputing unit is comprised in the display.
 8. A system according toclaim 1 in which the computing unit and the location measurement systemare comprised in the display.
 9. A system according to claim 1 in whichthe 3D image potentially appears floating in space, allowing a viewer tooptionally insert his/her hand or a tool into the 3D image space.
 10. Asystem according to claim 1 in which computing the data for displayingthe 3D image of the virtual object in the correct location relative tothe real object comprises registering a marker in the real object to amarker in the virtual object.
 11. A system according to claim 1 in whichcomputing the data for displaying the 3D image of the virtual object inthe correct location relative to the real object comprises registering aplurality of markers in the real object to a plurality of markers in thevirtual object.
 12. A system according to claim 1 in which the computingunit is adapted to compute data to display the 3D image of the virtualobject at least partly behind the surface of the real object based on aknown relative location of the virtual object with respect to a markingon the surface of the real object.
 13. A method for displaying athree-dimensional (3D) image of a virtual object in a correct locationrelative to a real object, comprising: measuring a location of a featureon the real object; registering a 3D image of the virtual object to bedisplayed to the determined location; and displaying the 3D image of thevirtual object in the correct location relative to the real object,wherein the displaying is adapted to display a plurality of differentpoints in the 3D image of the virtual object as focused at a pluralityof respective different distances from a viewer using the measuring. 14.A method according to claim 13, wherein the displaying comprisesdisplaying by a Head Mounted Display (HMD).
 15. A method according toclaim 14, wherein the determining the location on the surface of thereal object comprises using a location measurement unit comprised in theHMD.
 16. A method according to claim 14, wherein computing data fordisplaying the 3D image comprises using a computing unit comprised inthe HMD.
 17. A method according to claim 14, wherein the registering a3D image of a virtual object to be displayed to the location comprisesusing a computing unit comprised in the HMD.
 18. A method according toclaim 13 in which the registering comprises computing data fordisplaying the 3D image of the object in the correct location relativeto the real object based on measuring the location on the surface of thereal object.
 19. A method according to claim 18 in which the registeringcomprises computing data for displaying the 3D image of the object inthe correct location relative to the real object based on measuring alocation of a marker which has a three-dimensional asymmetric shape. 20.A method according to claim 13 and further comprising enabling a viewerto optionally insert his/her hand or a tool into the 3D image space.